Avermectin, as a broad-spectrum and highly effective antiparasitic drug, is commonly used to treat diseases in dairy cows and prone to remain in the body and can enter the food chain through milk, posing a threat to human health. In this study, a core-shell covalent organic framework composite (Fe₃O₄@TabDmtp) was prepared by a solvothermal self-assembly method using 1,3,5-tris(4-aminophenyl)benzene (Tab) and 2,5-dimethoxyterephthalaldehyde (Dmtp) as organic monomers. Fe₃O₄@TabDmtp was characterized by Fourier transform infrared spectroscopy, transmission electron microscopy and vibrating sample magnetometer. Fe₃O₄@TabDmtp was used as an adsorbent for the magnetic solid-phase extraction (MSPE) of avermectin, and the effects of eluent type and volume, adsorbent dosage, extraction time, pH and ionic strength on the extraction efficiency were investigated. Under the optimal extraction conditions, a new fluorescence method for the determination of avermectin was established. The calibration curve for avermectin exhibited good linearity in the concentration range of 0.02–10 μg/mL with a correlation coefficient of 0.998. The limit of detection and limit of quantification were 0.006 and 0.02 μg/mL at signal-to-noise ratio of 3 and 10, respectively. Recovery rates for spiked samples ranged from 98.36% to 114.76%. In conclusion, the pretreatment method is time saving and Fe₃O₄@TabDmtp can be reused, and the analytical method has high accuracy, strong stability and high resistance to matrix interference. It can be applied to the rapid detection of avermectin in whole, low-fat and skim milk samples.