Abstract: With the development and advancement of science and technology, the development and innovation of unmanned aerial vehicle (UAV) technology and products have brought great convenience to people in the fields of aerial photography, plant protection, electric cruise, and so on, but the development of UAVs also brings a series of management problems. Therefore, as a key part of the anti-UAV system, research into effective UAV detection is a pressing issue that must be addressed. In public environments such as parks, stadiums, and schools, the detection and tracking of UAV targets become more difficult due to their inherent characteristics and environmental factors. For example, under the occlusion of background interferences such as trees, buildings, and light, the target detection algorithm is unable to extract the effective features of the UAV target, resulting in target detection failure. It is of great significance to study the anti-occlusion target detection and tracking algorithm of anti-UAV systems for situations where UAVs cannot be successfully detected due to occlusion. This study proposes an improved anti-UAV system target detection algorithm YOLOX-drone based on YOLOX-S to solve the problem of the UAV being deformed and partially occluded in complex scenes, which makes it difficult to identify. First, in this study, numerous occluded drone images are collected in complex scenes, and the drone pictures are downloaded online for occlusion processing. The drone images were labeled to establish a UAV image dataset. Second, the YOLOX-S target detection network was constructed. On this premise, the coordinate attention mechanism is introduced to improve the saliency of the target image when the drone is obscured by highlighting useful features and suppressing useless ones. Then, the bottom-up path enhancement structure in the feature fusion layer is removed to reduce the network complexity, and an adaptive feature fusion network structure is designed to improve the expression ability of useful features, suppress interference, and improve detection accuracy. First, experiments were conducted on the Dalian University of Technology Anti-UAV dataset, and the experimental results show that YOLOX-drone improved average accuracy (IOU = 0.5) by 3.2%, 4.7%, and 10.1% compared to YOLOX-S, YOLOv5-S, and YOLOX-tiny, respectively. Then, experiments were conducted on the self-built UAV image dataset, and YOLOX-drone improved the average accuracy (IOU = 0.5) by 2.4%, 2.1%, and 6.4% in the cases of no occlusion, general occlusion, and severe occlusion, respectively, when compared with the original YOLOX-S target detection model. This demonstrates that the improved algorithm has good anti-occlusion detection ability.