I don't think it's so much that Free has no sense of direction, rather that he's just incompetent about all of his magic - spatial, ice magic, etc.

Ah! That makes sense. (Back in the Baba Yaga arc he ended up in the opposite tower to the one he was trying to find, but I don't remember how he got there.)