My prius was doing this a while back. It would also fluctuate in it's speed occasionally, which was annoying. It would only do it now & then, but then got to be more frequent. The fan assembly sits under the dash, in a position where it is almost like a trash receptacle for any dirt or dust that gets through the filter. So I took the fan blower out of the vehicle, and cleaned it out. First I blew out all of the dust, crud, and junk that collected in the assembly, then lubricated the shaft liberally. That was several thousand miles ago,and it works perfect now.
It's an easy task, and worth trying. Now, if you are having issues with the motor not working for certain speeds, it would be more likely an issue with the relay, but try cleaning it first.