"I then have 4 System actions:
Set b to random(254)
Set g to random(254)
Set r to random(254)
Set layer "background" background color to rgb(r,g,b)
But when I run the game the background does not change."
Maybe using floor(random(254)) for each, since the Random function will not return only integers(in fact, they have almost no chance of returning an integer)
Also, check that your layer is not transparent
And that is why you shall respect the bug reports guidelines, not only giving a capx is making the bug reproductible in one click in a situation they can work with (less time wasted trying to reproduce vague instructions) but also it helps filtering false positives.
Game design is all about decomposing the core of your game so it becomes simple instructions.