Is Cosine-Similarity of Embeddings Really About Similarity?

Original Article ↗ Hacker News Discussion ↗

Role of Cosine Similarity in Semantics

  • Many argue cosine similarity on sentence/word embeddings has little true “semantic” understanding; it often confuses antonyms with synonyms because they share contexts.
  • Others counter that meaning comes from the embedding model and training objective; cosine is just a geometric tool. If embeddings are trained for semantic similarity (e.g., contrastive losses), cosine can be very effective.
  • Some claim it is “semantically vacuous” and mainly useful for information retrieval, not deep language understanding.

Dependence on Embedding Construction

  • Several comments stress that similarity behavior is entirely determined by how embeddings are learned, not by cosine itself.
  • Using cosine on models that were never trained with a cosine-based or similarity-aware loss is criticized as unjustified; the paper’s examples are seen as illustrating this mismatch.
  • Normalization is a recurring theme: for L2-normalized vectors, cosine similarity, dot product, and (transformed) Euclidean distance induce essentially the same rankings.

Alternatives and Distance Metrics

  • Suggested alternatives: Euclidean (L2), L1, squared differences, KL divergence, Maximum Mean Discrepancy, “Gaussian annulus” losses, and domain-specific ecological metrics.
  • Some note that there is an entire mathematical and ecological toolbox of distances, and that context-specific or manifold-aware metrics may be more appropriate than a flat dot product.
  • Others insist there is no fundamentally better general-purpose “cheap” replacement; trade-offs are between speed, dimensionality, and task-specific performance.

Practical Use Cases and Benchmarks

  • Despite theoretical criticism, embeddings plus cosine are said to work well for bitext mining, classification, clustering, paraphrase detection, and reranking, as reflected in benchmark suites.
  • There is skepticism toward leaderboard-driven evaluation and claims that current NLP has drifted into pure information retrieval.

Linguistic and Geometric Critiques

  • Some argue that latent vector spaces are the wrong formalism for core linguistic meaning, advocating logical/categorial grammar approaches instead.
  • Others highlight manifold curvature and non-uniform density of embedding spaces, warning that Euclidean/cosine distances may be locally or globally misleading.