On the Proposed California SB 1047
California’s proposed SB 1047 would impose safety, reporting, and shutdown requirements on developers of large AI models, prompting intense debate over whether it meaningfully reduces risk or simply entrenches big tech. Critics warn that compute-based thresholds, paperwork, and potential liability will crush smaller players, push work out of state, and regulate abstract “AI” rather than concrete harmful applications like fraud or defamation. Supporters counter that lessons from social media and other unregulated technologies justify proactive guardrails and public infrastructure such as a state-funded compute cluster to keep AI development accountable and more broadly accessible.
Scope of Regulation: Models vs Applications
- Strong split between regulating large foundation models vs regulating downstream applications/automated decision systems.
- Critics say SB 1047’s focus on compute/benchmarks favors incumbents, is easy to game, and pushes small players onto big-company APIs, creating an oligopoly.
- Others argue what matters is systems that affect people’s rights and opportunities (loans, employment, etc.), which existing data‑protection frameworks already partly cover.
Risk Framing and Existential Threats
- Some view rogue/“Terminator-style” AI as overblown relative to real human actors and argue AI “does not want” anything.
- Others think that if developers themselves talk about world‑ending risk, that alone justifies serious oversight.
- The bill is seen as mainly targeting rogue-model risk, not user-requested misuse, which some find conceptually inconsistent.
Liability, Misuse, and Deepfakes
- Debate over whether developers or hosting firms should be liable when tools are used for fraud, identity theft, or deepfake porn.
- One side: tools are general-purpose like cars/knives/Photoshop; punish fraud and defamation, not toolmakers.
- Other side: foreseeable, preventable harms (e.g., training on celebrity images and not blocking nude deepfakes) should create corporate liability, similar to unsafe car design.
- Concerns that relying on lawsuits alone is unrealistic because litigation is costly compared to creating harm.
Competition, Bureaucracy, and Thresholds
- Worry that compliance “paperwork,” safety determinations, and ambiguous model thresholds will be manageable only for big tech, entrenching them.
- Counterpoint: regulation can also keep markets open via antitrust-style constraints; need suspicion of both over‑ and under‑regulation.
- The bill’s model-definition via FLOPs and “state of the art” benchmarks is seen as vague, potentially covering far more than a few frontier LLMs and even non‑LLM models.
CalCompute Public Cluster
- Some see a public compute cluster as critical to democratize access, enable open research, and counter big‑tech dominance.
- Others fear a procurement boondoggle: captured by existing contractors, slow, obsolete on arrival, and potentially politicizing or centralizing control over open work.
California Context and Timing
- Disagreement on whether heavy AI rules will drive companies and talent out of California vs whether firms will stay because of ecosystem advantages.
- Some argue regulation is “way too early” and should wait for clearer harms; others say social media shows the cost of waiting and that we should regulate foreseeable harms now, iteratively.