I already wrote an object tracking algorithm that is quite fast, however that algorithm uses spatial proximity between points to determine whether or not a group of points are all part of the same object. That is, if a set of points are sufficiently close together, then they’re treated as part of one object. Then, that object is tracked as it moves.
It just dawned on me that you could also track an object by looking at the motions of a set of points. For example, if you have two objects, one of which is stationary, the other of which is moving, then the point data for those objects will reflect this. You can calculate the velocity of the points by using the nearest neighbor method to map each point in frame 1 to a point in frame 2, and so on and so on (this is exactly how my earlier object tracking algorithm works). You could then look at the change in position between each frame, and assuming each frame is taken over a uniform amount of time, that change in position is equal to the velocity of the point.
You would then cluster the points using their velocities, which would cause in this case the points in the stationary object to be clustered together, since its points are not moving, and cause the points in the moving object to be clustered together, since they’re all moving at roughly the same velocity.