Dot Product

Understanding Transformers Part 6: Calculating Similarity Between Queries and Keys

This article details the calculation of similarity between queries and keys in Transformers using the dot product, illustrating how a word's similarity to itself is higher than to other words. It explains that these scores are then transformed into meaningful weights via a softmax function.