"I then have 4 System actions:
Set b to random(254)
Set g to random(254)
Set r to random(254)
Set layer "background" background color to rgb(r,g,b)
But when I run the game the background does not change."
Maybe using floor(random(254)) for each, since the Random function will not return only integers(in fact, they have almost no chance of returning an integer)
Also, check that your layer is not transparent
Game design is all about decomposing the core of your game so it becomes simple instructions.