Irregularly shaped small fields are widely distributed in the hilly farmlands. Their curved and undulating headland boundaries have also posed persistent challenges for autonomous navigation. Unmanned edging operations are often required for the robust perception of complex field structures. Smooth, safe obstacle avoidance paths can be expected to maintain the unilateral guidance along the inner headland boundary. Conventional techniques of machine vision cannot distinguish the subtle texture differences between headland and farmland under varying field conditions. Low robustness has also limited spatial patterns in recent years. Likewise, conventional local path planning cannot provide for the smooth obstacle-bypassing and a stable, consistent rejoining of the reference boundary suitable for unilateral guidance scenarios. In this study, a path planning was proposed for the unilateral guidance obstacle avoidance using RGB-D multimodal data. Two components consisted of: high-fidelity scene mapping and real-time path planning. At the perception level, a depth camera was used to simultaneously capture color and depth information. An efficient RGB-D semantic segmentation model was adopted to exploit the complementary RGB texture and depth geometry cues. The network followed a dual-stream encoder–decoder design with multilevel cross-modality fusion, attention-based feature reweighting, and multi-scale supervision. Accurate pixel-level classification of headland and obstacle regions was realized under diverse lighting, crop residue, and soil conditions. Depth information also enhanced separate cases, where RGB contrast was weak, such as dry field headland edges, indicating stable segmentation over different terrains. The segmentation was then transformed into spatially meaningful representations. A three-stage processing pipeline was developed for the point cloud. (1) Each RGB pixel was back-projected into 3D space using depth measurements and camera calibration parameters for a dense point cloud. (2) Semantic labels were transferred from the RGB-D segmentation to the 3D points using pixel-point indexing, thus resulting in a semantically annotated point cloud. (3) A ground-plane estimation and projection procedure was used to generate a 2D obstacle-avoidance coordinate map, where all relevant points were projected onto a common field plane. The coordinate map provided a compact yet informative representation for efficient, real-time path planning. An enhanced artificial potential field algorithm was designed to meet the single-edge-guided obstacle avoidance. Several improvements were introduced, including geometric generalization of repulsive fields to handle arbitrarily shaped obstacles, region-aware gain modulation to regulate repulsive forces near the headland boundary, integration of a smoothing factor to eliminate curvature discontinuities, headland-specific repulsion modelling to maintain a stable lateral offset from the boundary, and sub-target generation to avoid local minima caused by force equilibrium. The planner generated obstacle avoidance trajectories with a gentle obstacle approach, early exit, and overall path smoothness, while maintaining the optimal distance from the headland. Path tracking controllers were also selected via machine vision. The planned trajectory was further re-projected onto the forward-looking RGB image using inverse coordinate transformation. The downstream control module then visually aligned path references. Extensive experiments were conducted to validate the effectiveness of the framework. The RGB-D segmentation network achieved the highest mean Intersection over Union of 95.97% on the paddy-field and dry-field datasets, thus producing accurate and stable segmentation. The potential field planner also generated obstacle avoidance trajectories with path deviation below 9 pixels and standard deviations under 1.2 pixels over various obstacle shapes and placements. Furthermore, field experiments demonstrated that better performance was achieved in an average lateral deviation below 7.6 cm during operation. Overall, the framework can provide an integrated solution for autonomous obstacle avoidance under unilateral headland guidance. The reliable navigation can be combined with multimodal perception, 3D spatial reasoning, and behavior-aware planning. A practical approach can also contribute to fully unmanned operations on irregularly shaped small fields in smart agriculture.