High-Altitude Long-Endurance (HALE) solar-powered Unmanned Aircraft Vehicles (UAVs) can utilize solar energy as power source and maintain extremely long cruise endurance, which has attracted extensive attentions from researchers. Trajectory optimization is a promising way to achieve superior flight time because of the finite solar energy absorbed in a day. In this work, a method of trajectory optimization and guidance for HALE solar-powered aircraft based on a Reinforcement Learning (RL) framework is introduced. According to flight and environment information, a neural network controller outputs commands of thrust, attack angle, and bank angle to realize an autonomous flight based on energy maximization. The validity of the proposed method was evaluated in a 5-km radius area in simulation, and results have shown that after one day-night cycle, the battery energy of the RL-controller was improved by 31% and 17% compared with those of a Steady-State (SS) strategy with a constant speed and a constant altitude and a kind of state-machine strategy, respectively. In addition, results of an uninterrupted flight test have shown that the endurance of the RL controller was longer than those of the control cases.