In future 6G wireless communications, extremely large-scale antenna arrays and high-frequency bandwidths are expected to play a crucial role. Their utilization increases the likelihood that communication devices operate in the near-field region, where traditional transceiver architectures suffer from broadband beam-splitting effects that signifcantly degrade communication performance. To allieviate this issue, this paper studies the utilization of unique frequency selective properties of dynamic metasurface antennas (DMAs) in near-field uplink transmission. Specifically, we aim to configure the tunable parameters of the DMA to maximize the average beamforming gain across all subcarriers. Morever, we propose a deep reinforcement learning framework to efficiently learn the optimal tunable parameters of the DMA. Simulation results demonstrate that the proposed scheme achieves better beamforming gain performance over all subcarriers compared with existing benchmarks.