You can make a normal rectangle the limit with clamp():
set position clamp(self.x, left,right), clamp(self.y, top,bottom)
So to do a rounded rectangle as the limit do something like this:
1. limit to a rectangle first
2. check if object is in a corner of the rectangle.
3. if it is then limit it to a circle of that corner.