Learn English – “Would not pass” or “would not have passed”

Here is a situation:

(Jack and Dorothy are in the car)

Silence. They pass Blumergton. Dorothy looks out of the window, her face is obviously surprised.

Silence again. They pass Himilgreens.

Silence again.

Jack: "I must tell you something"

Dorothy: "What is it?"

Jack: "We are not going to Darwinhill"

Dorothy: "I know that"

Jack: "How come you know that?"

Dorothy: "Otherwise we would've not passed Blumergton"

or should it be "Otherwise we would not pass Blumergton" or should it be some other way?