What is the origin of the Vanir?
Big question. Let's start from the start:
The mythological origin of the Vanir
This is a big question mark - there is nothing in the myths about the world's creation that suggests anything about where the Vanir came from. There is no good hint at all. The Vanir seems to be somehow related to the elves - probably the Ljósálfar ("light elves"), and the Dökkálfar ("dark elves") are by some identified with the dwarves, for which we have creation myths - but this is an extreme long shot, and trying to make something of it would not make much sense at all.
The Vanir suddenly appear in the mythology. We are told that they hail from Vanaheim, and the first one we meet in Völuspa is a strange Völva called Gullveig, whom the Aesir spears and attempts to burn three times, but each time she is reborn. It is possible that she is really Freya under another name. After this, the great war between the gods begins; a war that ends in a draw and a peace between the gods, where they exchange hostages. There is a longer about what happened after the end of the war, but for this discussion, the important end result was that the Aesir received Njord, Frey and Freya. This is all attested in Völuspa and by Snorri in Heimskringla.
The real-world history of the Vanir
Here we have something extremely rare in the field of norse mythology: a very early attestation of worship, in the form of an account in Tacitus' Germania. Tacitus tells us of the goddess Nerthus, primarily of how she was worshipped in southern Jutland. Still, just her name has given quite a bit of insight, as it is clearly a cognate of "Njord". A goddess *Njärd can also be found in some place names. Scholarly speculation has been along two main lines:
- That the goddess Nerthus somehow evolved into the god Njord.
- That this is a reflection of a custom of the Vanir the myths tells us about: the sibling marriage. Just as brother and sister Frey and Freya shares the stem of their name, so would Njord and Nerthus. Nerthus would thus be the mother of Frey and Freya, but would have faded into nothing by the time the Norse myths were written down.
Finally, it should be noted that the Vanir seems to have been an exclusively Scandinavian set of gods: there are no attestations of them in continental Germanic sources.
The role of the Vanir
After the war, and some disputes following it, the Aesir and Vanir seem to have gotten along well; they were worshipped at the same places, and there is no reason to suppose there were any friction between those that preferred one god to another. (Margaret Clunies Ross have a different interpretation, where the Aesir would be dominant over the Vanir, but I am not convinced that there is sufficient evidence of this, as the number of Vanir appearing in the myths is so small).
In terms of roles, the Vanir were associated with things like fertility, harvests, and bounty from the sea (this is true even of Nerthus, who seems to have been a Great mother type of goddess). Freya was an especially important deity: she had first pick of half the dead heroes for her hall, where the remainder went to Odin and Valhall. She was also responsible for teaching the Aesir the use of Seiðr, a potent form of magic, even if it was considered unmanly. And while Frey is primarily a god for farmers, his name means "ruler", and he seems to have had a warlike side as well. He was also thought to have been the primogenitor of the old Swedish royal house.
Apart from the primary sources indicated in the text, I have mostly relied on Gro Steinsland, Fornnordisk religion. For another take on the relative status of Aesir and Vanir, and a more in depth discussion of their war, see Margaret Clunies Ross, Prolonged echoes.
If I were to make an educated guess, I'd say there was some syncretism going on, and the war between the Aesir and Vanir was a memory of the struggles between two peoples, one whose gods were Odin and company, and the others who worshipped Njord/ Frey etc. And IMO, the limted geographic extent of the Vanir reinforces this idea.