How does cosine similarity work?

Original Article ↗ Hacker News Discussion ↗

Cosine similarity—effectively a dot product between length‑normalized vectors—is widely used to compare text and other embeddings, but its intuition and limitations are often misunderstood. Commenters explore why it works well for bag‑of‑words and modern NLP embeddings, how it relates to geometry, correlation, and Euclidean distance, and why magnitudes are often ignored by convention. They also highlight practical issues such as performance and numerical stability in libraries, high‑dimensional effects, cases where other distance metrics are more appropriate, and the importance of matching the similarity measure to how an embedding model was trained.

Practical implementation notes

  • Cosine distance is available in common libraries, but users report SciPy’s distance module as slow, prone to overflow in mixed precision, and using suboptimal math functions; it’s fine for small datasets but not “big data.”
  • For JavaScript, multiple passes over standard arrays are seen as costly; TypedArray plus simple loops are recommended for speed and compatibility with native extensions.

Why cosine similarity is popular (especially in NLP)

  • In classic information retrieval, cosine on bag‑of‑words vectors naturally implements “length‑normalized word counting,” avoiding bias toward longer documents.
  • With word or sentence embeddings, cosine is interpreted as “how much the same features are active in both,” while ignoring overall scale.
  • Some see it as essentially “normalized dot product” and prefer that framing; others emphasize its geometric meaning as the cosine of the angle between vectors.

Normalization vs magnitude

  • Many practitioners normalize embeddings to unit length so cosine becomes a dot product; they ask whether magnitude ever matters.
  • Replies note that models (e.g., language models’ logits) use unnormalized dot products, so magnitude can encode information like token frequency.
  • Some applications use L2 distance where magnitude strongly affects similarity.
  • Debate arises over whether normalization “loses a dimension” or just discards magnitude information; the geometry of unit spheres and manifolds is discussed.

High-dimensional behavior

  • Multiple comments note that in high dimensions, random vectors tend to have cosine near zero and distances cluster, but cosine still works well for relative ranking.
  • There’s mention of work suggesting not normalizing in high‑dimensional ML spaces and of the general “curse of dimensionality,” though details are left vague.

Geometry vs abstraction

  • One camp insists we are doing geometry/trigonometry (angles, projections, spheres) and finds that intuitive, even in high dimensions.
  • Another camp prefers viewing vectors as feature lists or functions where cosine is just an inner product–based correlation, arguing that geometric imagery can mislead beyond 3D.

Alternatives and criticisms

  • Cosine similarity is criticized as status‑quo and sometimes misapplied, especially when vector magnitude carries important meaning or data include negatives, noise, or temporal/geospatial structure.
  • Suggested alternatives include Euclidean, L1/Manhattan, Chebyshev, Jaccard, Pearson correlation, Hamming, and problem‑specific metrics.
  • It’s noted that if all vectors have equal norm, cosine similarity and Euclidean distance induce the same nearest‑neighbor ordering.