We introduce CADSpotting, an effective method for panoptic symbol spotting in large-scale architectural CAD drawings. Existing approaches often struggle with symbol diversity, scale variations, and overlapping elements in CAD designs, and typically rely on additional features (e.g., primitive types or graphical layers) to improve performance. CADSpotting addresses these challenges with a primitive-agnostic, coordinate-only dense sampling representation: each CAD primitive is converted into densely sampled 2D points embedded in 3D with z = 0, and a point-cloud backbone learns features without explicit primitive-type or layer attributes. To enable accurate segmentation in large drawings, we further use sliding window aggregation (SWA), which combines weighted voting and sparse non-maximum suppression (NMS) to merge local predictions across overlapping windows. Moreover, we introduce LS-CAD, a large-scale dataset comprising 45 finely annotated complete floorplans, each covering approximately 1000 m² or more. LS-CAD complements existing large-quantity CAD datasets by focusing on physically extensive real-world layouts with thousands to tens of thousands of primitives per drawing. Experiments on FloorPlanCAD and LS-CAD demonstrate that CADSpotting achieves strong performance compared to existing methods. We also showcase its practical value in enabling automated parametric 3D interior reconstruction directly from raw CAD inputs.