The killer app of Gemini Pro 1.5 is using video as an input
Google’s Gemini Pro 1.5 model can accept long video inputs by sampling them as image frames (roughly one frame per second at ~258 tokens each), enabling tasks like indexing a bookshelf from a short phone video rather than manual data entry or traditional OCR. Commenters compare it to GPT‑4 Vision, probe how video and image tokens really work, and debate accuracy, hallucinations, cost, and the practicality of using such systems for moderation or automation. A major undercurrent is concern over privacy and surveillance: once video streams can be cheaply understood at scale, everything from personal “memory assistants” to state and corporate panopticons becomes far more feasible.
How Gemini 1.5 Handles Video
- Video is effectively treated as a sequence of images; docs and report indicate ~1 frame/second is extracted, with audio dropped.
- A 45‑minute film example in the report: 2,674 frames at 1 FPS ≈ 684k tokens; per‑frame cost matches single‑image token usage.
- Several commenters confirm token math: ~258 tokens per image, ~1,800 tokens for a 7s clip ⇒ ~7 frames ≈ 1 FPS.
- There’s mention of scene segmentation and indexing for longer videos, but details are tooling‑level and not fully described.
Tokens, Multimodal Encoding, and Confusion
- Multiple people struggle with how “tokens” apply to images/video.
- One camp argues images are converted into descriptive text before entering the LLM; another points to Google papers (Flamingo, PaLI, ViT‑style models) saying images are tokenized as visual patches, not via an external OCR pass.
- Consensus: billing is expressed in “tokens” for user familiarity, but underlying image tokens are not the same as text tokens; exact internal mapping is unclear.
Comparison to GPT‑4V and Classic OCR
- Report claims Gemini 1.5 can process up to an hour of video at 1 FPS, whereas GPT‑4V reportedly errors around 3–4 minutes at 1 FPS.
- Per‑frame token cost appears lower than GPT‑4V’s at high detail.
- Many note that bookshelf indexing is essentially fancy OCR plus long context. Supporters say the novelty is scale + multimodal reasoning; skeptics say similar pipelines (frames + OCR + LLM) are already possible with other tools.
Use Cases and Applications
- Suggested uses: indexing personal bookshelves, video understanding for analytics, content moderation, traffic/plate recognition, workflow assistants that “watch” your screen, posture monitoring, language learning, “perfect memory” personal logs.
- Some see this as a “killer app” for turning messy real‑world visuals into structured data quickly.
Safety, Moderation, and Censorship
- Strong criticism of over‑aggressive safety filters (e.g., blocking “cocktail,” sexual terms, some historical images).
- Debate over whether such guardrails are necessary versus offering less‑filtered modes.
- Prompt‑injection via video is flagged as a real concern.
Privacy, Surveillance, and Societal Impact
- Extensive worry that cheap, scalable video understanding destroys practical privacy and supercharges state and corporate surveillance (protests, consumer profiling, law enforcement).
- Some accept more surveillance for safety (e.g., traffic enforcement); others argue this undermines democracy and protest.
Skepticism and Limitations
- Errors in extracted book titles and hallucinations reduce trust for archival/mission‑critical uses; many see LLMs as accelerators requiring human review.
- Debate over whether current LLMs are toys vs. serious automation tools; several note they can already replace narrow human tasks at scale despite non‑zero error rates.