I’ve been reading Daniel Pink’s book Drive, and thinking about intrinsic and extrinsic motivation. In particular, Rollercoaster games rely heavily on carrot/stick motivation (extrinsic rewards and punishments), and Experiment and Mastery games more on Autonomy, Purpose, and Mastery (his tools to create intrinsic motivation).
This is really interesting, because it provides a clue as to why these genres have evolved the way they have. Since Rollercoaster games rely heavily on rewards and punishments, they have to be short (extrinsic rewards don’t motivate for very long), and rely heavily on giving you new and exciting rewards (since you get acclimated to them quickly), which story is perfect for. Story also gives you very strong Purpose, above the gameplay, which probably helps sustain play over hours despite the limited Autonomy and Mastery. They are happy to add story even if though it usually hurts gameplay.
Meanwhile, Experiment and Mastery games have deep Autonomy and Mastery, and so need less of these extrinsic rewards to get players to heavily invest. They don’t have to make hard choices between story and gameplay. Gameplay alone is enough.