Wish, if only, it’s time, would rather

English uses the past tense for two quite different jobs: real past time, and unreality. After certain expressions, only the second job is in play.

For a present situation you want to be different, use the past simple: I wish I knew, if only he were here.

For a past situation you regret, go one step further back into the past perfect: I wish I had called her, if only we had left earlier.

For someone else’s behaviour that irritates you, use would: I wish you would stop doing that. You cannot use would about yourself in this pattern.

It’s time and would rather follow the same logic: it’s time we left, I’d rather you didn’t.