TY  - JOUR
AU  - Yi, Wenfeng
AU  - Wu, Wenhan
AU  - Chen, Maoyin
AU  - Zheng, Xiaoping
PY  - 2025
TI  - Human Morality Difference when Programming and Actually Operating Autonomous Machines
JO  - Tsinghua Science and Technology
SN  - 1007-0214
SP  - 1648
EP  - 1658
VL  - 30
IS  - 4
AB  - Autonomous machines (AMs) are poised to possess human-like moral cognition, yet their morality is often pre-programmed for safety. This raises the question of whether the morality intended by programmers aligns with their actions during actual operation, a crucial consideration for a future society with both humans and AMs. Investigating this, we use a micro-robot swarm in a simulated fire scenario, with 180 participants, including 102 robot programmers, completing moral questionnaires and participating in virtual escape trials. These exercises mirror common societal moral dilemmas. Our comparative analysis reveals a “morality gap” between programming presets and real-time operation, primarily influenced by uncertainty about the future and heightened by external pressures, especially social punishment. This discrepancy suggests that operational morality can diverge from programmed intentions, underlining the need for careful AM design to foster a collaborative and efficient society.
UR  - https://doi.org/10.26599/TST.2024.9010062
DO  - 10.26599/TST.2024.9010062