They need some help figuring out which words go together and what order they should be in. That’s where relative positional embeddings come in!

Basically, these little guys add extra information to each word that tells the computer where it is in relation to other words on the page. For example, if you have a sentence like “I ate pizza for lunch,” your computer might see something like this:

[1] I [2] ate [3] pizza [4] for [5] lunch [6].

Each number represents a word in the sentence and its position relative to other words. So, when the computer reads “for” (word 5), it knows that it’s coming after “ate” (word 3) and before “lunch” (word 6). This helps the computer understand the context of each word and how they relate to one another.

Now, let’s say you have a really long text document with lots of different sentences. Your computer might struggle to keep track of all those words and their positions relative to other words on the page. That’s where these fancy new embeddings come in! They allow your computer to learn from unlabeled documents (meaning there are no right or wrong answers) by using an area-masking strategy. This means that instead of trying to figure out which sentences are correct, your computer can focus on learning how words relate to each other based on their positions relative to one another.

So, if you have a sentence like “I went to the store and bought some groceries,” your computer might see something like this:

[1] I [2] went [3] to [4] the [5] store [6] and [7] bought [8] some [9] groceries.

Each number represents a word in the sentence, but now there are extra numbers inside brackets that tell your computer which words go together based on their positions relative to each other. For example, “to” (word 4) and “the” (word 5) both belong to the same group because they’re next to each other in the sentence. This helps your computer understand how different groups of words relate to one another and what order they should be in.

Overall, relative positional embeddings are a pretty cool way for computers to learn from unlabeled documents by focusing on learning how words relate to each other based on their positions relative to one another. And the best part is that it’s all done without any fancy math or complicated algorithms!