Have you double checked that their collision boxes are set to rectangles? Sometimes it will try and guess the shape and make it some kind of polygon.
Right click one of the collision box points and choose "set to bounding box"
If that doesn't, depending on how your game works, you might need to overwrite its vertical movement to force it to do only what you want.
Sounds like its colliding into the other objects and when two solid objects are inside one another one of them will jump as you've described it.
can you post a .capx for us to take a look at?