Pilot Selecting Preferences: Based on a Textual Analysis of Central Government Policy Documents

The selection of pilots, a core stage in the policy experimentation process, affects the innovation and diffusion of the experiment, and any sampling bias may negatively affect the representativeness and applicability of the policy experiments. Because researchers have few opportunities to directly observe or interview the decision-makers in the policy selection process, there has been limited research on the selection of pilots. Surveys are particularly difficult to conduct for large-N studies that would enable researchers to compare and refine the characteristics and logic behind the selection of pilots. This study regards policy documents on policy experimentation as official statements that reveal the preferences and required standards of the decision-makers. Official published statements provide an alternative perspective to look at the biases and considerations of the decision-makers when they design policy experiments and select pilots. We have collected policy documents on 385 policy experiments during the 13th Five-Year Plan and then have created a pilot city dataset on the first-round list of experiments. A Chinese word frequency analysis of the content of these related policy documents then looked for important factors that may influence selection of the pilots. The study yields three findings. First, pilots are concentrated in national and regional central cities, and the geographical distribution of the pilots among 336 cities in China is highly uneven. Second, word frequency co-occurrence analyses show that decision-makers overwhelmingly prefer cities with rich work experience and better conditions. Last but not least, a cross-policy-domain study shows that the preferences and concerns of decision-makers vary depending on the different policy areas. Based on keywords, we have developed six dimensions to measure the decision-makers' priorities in terms of selection of the pilots. These include the pre-work foundation, willingness, implementation capacity, representativeness, ability to innovate, and strategy-oriented thinking. Economic policy experiments focus on the innovative abilities and strategic importance of the cities, and therefore this category of policy experiments is concentrated in cities with a higher level of economic performance. However, major concerns of agricultural and rural policy experiments are their representativeness. At the beginning of the policy experiments, decision-makers also consider the regional diversity and the adaptability for experimentation. In terms of environmental policy, the representativeness and strategic importance of the cities are of equal importance to the decision-makers. The findings suggest decision-makers follow different preferences and logics in selecting pilots according to the policy goals, the environment, and the implementation conditions for specific policy experiments. Each policy domain has special and unique considerations and requirements during the period of selecting the pilots. We propose that the policy domain is an important element that explains the diverse logics behind the selection of pilots.