Ah, I see the confusion now. You've missed the 'initial visibility' property :)
Layers have two properties:
Initial visibility - only affects the runtime
Visible in editor - only affects the editor
The checkbox is a shortcut for the "visible in editor" option. So if you want to change the runtime use the "initial visibility" option!