@Excal, I don't have real devices with touch to test it, but I'll work a little bit over this file to give you a better approach.
Maybe it can be a bug, and you also reported it, that's why I shut up my mouth before.
@Excal, the best I can do while checking with mouse is there.
What I did is store the lastX for the first and second touch IDs, because I'm using mouse, when I press the second button, the first stop to work, so, the first X stop to update.
but, it is using the touch index, as you want.
Hope that helped.