Halliday & Matthiessen (2004: 544-5):
Note however that most apparently spatial cohesion is in terms of metaphorical space, for example, there in there you’re wrong; cf expressions like on those grounds, on that point. These are actually expressions of Matter. Many conjunctive expressions of the expanding kind are also in origin spatial metaphors; for example, in the first place, on the other hand (hand involves a double metaphor: ‘part of the body’ — ‘side’ [on my right hand] — ‘side of an argument’).