Historians often portray the human settlement of Tamriel as a straightforward process of military expansion of the Nords of Skyrim. In fact, human settlers occupied nearly every corner of Tamriel before Skyrim was even founded. These so-called "Nedic peoples" include the proto-Cyrodilians, the ancestors of the Bretons, the aboriginals of Hammerfell, and perhaps a now-vanished Human population of Morrowind. Strictly speaking, the Nords are simply another of these Nedic peoples, the only one that failed to find a method of peaceful accommodation with the Elves who already occupied Tamriel.
Ysgramor was certainly not the first human settler in Tamriel. In fact, in "fleeing civil war in Atmora", as the Song of Return states, Ysgramor was following a long tradition of migration from Atmora; Tamriel had served as a "safety valve" for Atmora for centuries before Ysgramor's arrival. Malcontents, dissidents, rebels, landless younger sons, all made the difficult crossing from Atmora to the "New World" of Tamriel. New archeological excavations date the earliest human settlements in Hammerfell, High Rock, and Cyrodiil at ME800-1000, centuries earlier than Ysgramor, even assuming that the twelve Nord "kings" prior to Harald were actual historical figures.
The Nedic peoples were a minority in a land of Elves, and had no choice but to live peacefully with the Elder Race. In High Rock, Hammerfell, Cyrodiil, and possibly Morrowind, they did just that, and the Nedic peoples flourished and expanded over the last centuries of the Merethic Era. Only in Skyrim did this accommodation break down, an event recorded in the Song of Return. Perhaps, being close to reinforcements from Atmora, the proto-Nords did not feel it necessary to submit to the authority of the Skyrim Elves. Indeed, the early Nord chronicles note that under King Harald, the first historical Nord ruler (1E 113-221), "the Atmoran mercenaries returned to their homeland" following the consolidation of Skyrim as a centralized kingdom. Whatever the case, the pattern was set -- in Skyrim, expansion would proceed militarily, with human settlement following the frontier of conquest, and the line between Human territory and Elven territory was relatively clear.
But beyond this "zone of conflict", the other Nedic peoples continued to merge with their Elven neighbors. When the Nord armies of the First Empire finally entered High Rock and Cyrodiil, they found Bretons and proto-Cyrodiils already living there among the Elves. Indeed, the Nords found it difficult to distinguish between Elf and Breton, the two races had already intermingled to such a degree. The arrival of the Nord armies upset the balance of power between the Nedic peoples and the Elves. Although the Nords' expansion into High Rock and Cyrodiil was relatively brief (less than two centuries), the result was decisive; from then on, power in those regions shifted from the Elves to the Humans.