The words ‘former’ and ‘latter’ literally mean ‘first-er’ (from Old English forma, first) and ‘late-er’ (from Old English lætra, later).
So why do we pronounce ‘later’ and ‘latter’ differently today?
In many monosyllabic Old English words with a short æ vowel (like fæt, dæġ, hwæl), a phonological process called ‘retraction’ occurred in some inflected forms, changing the /æ/ to /ɑ/. So the plural of OE hwæl (whale) was hwalas, and the plural of fæt (container) was fatu.
This happens in the comparative adverb of læt, being lator, but not in the comparative adjective, being lætra, so already in the Old English period the vowels were different.
Predictably, lator is now ‘later,’ pronounced with the normal vowel we would expect from OE /ɑ/ (compare nama -> name; bacan -> bake), and lætra is now ‘latter,’ pronounced with the normal vowel we would expect from OE /æ/ (compare cætt -> cat; æsc -> ash).