Is Cosine-Similarity of Embeddings Really About Similarity?

Original Article ↗ Hacker News Discussion ↗

Cosine similarity, the standard way to compare high‑dimensional embeddings in machine learning, is questioned for its ability to reflect true semantic similarity, especially when models weren’t explicitly trained for that objective. Commenters note that embeddings can place antonyms close together and that cosine scores are invariant to arbitrary rotations or rescalings, making them potentially misleading without task-specific training. Alternatives such as Euclidean distance, KL divergence, MMD, or domain-specific metrics are mentioned, but there is no consensus on a clearly superior, computationally cheap replacement.

Role of Cosine Similarity in Semantics

  • Many argue cosine similarity on sentence/word embeddings has little true “semantic” understanding; it often confuses antonyms with synonyms because they share contexts.
  • Others counter that meaning comes from the embedding model and training objective; cosine is just a geometric tool. If embeddings are trained for semantic similarity (e.g., contrastive losses), cosine can be very effective.
  • Some claim it is “semantically vacuous” and mainly useful for information retrieval, not deep language understanding.

Dependence on Embedding Construction

  • Several comments stress that similarity behavior is entirely determined by how embeddings are learned, not by cosine itself.
  • Using cosine on models that were never trained with a cosine-based or similarity-aware loss is criticized as unjustified; the paper’s examples are seen as illustrating this mismatch.
  • Normalization is a recurring theme: for L2-normalized vectors, cosine similarity, dot product, and (transformed) Euclidean distance induce essentially the same rankings.

Alternatives and Distance Metrics

  • Suggested alternatives: Euclidean (L2), L1, squared differences, KL divergence, Maximum Mean Discrepancy, “Gaussian annulus” losses, and domain-specific ecological metrics.
  • Some note that there is an entire mathematical and ecological toolbox of distances, and that context-specific or manifold-aware metrics may be more appropriate than a flat dot product.
  • Others insist there is no fundamentally better general-purpose “cheap” replacement; trade-offs are between speed, dimensionality, and task-specific performance.

Practical Use Cases and Benchmarks

  • Despite theoretical criticism, embeddings plus cosine are said to work well for bitext mining, classification, clustering, paraphrase detection, and reranking, as reflected in benchmark suites.
  • There is skepticism toward leaderboard-driven evaluation and claims that current NLP has drifted into pure information retrieval.

Linguistic and Geometric Critiques

  • Some argue that latent vector spaces are the wrong formalism for core linguistic meaning, advocating logical/categorial grammar approaches instead.
  • Others highlight manifold curvature and non-uniform density of embedding spaces, warning that Euclidean/cosine distances may be locally or globally misleading.