Note the Horse's Body Language

Note the direction the horse’s ears are pointing. A horse’s ears can move independently of one another, as can its eyes. The direction the ear is pointing indicates the direction the eye on that side of its head is looking.

If the horse is looking at you and its ears are pointing in your direction, it is probably not considering which way to bolt.