SB-1047 will stifle open-source AI and decrease safety

Meta: Article Quality and AI-Generated Content

• Several commenters dismiss the article as low‑quality and possibly LLM‑generated, citing vague claims, overuse of bullet points, and lack of direct bill citations.
• Others argue that speculating about AI authorship is a distraction and that voting/flagging should be based on substance, not on perceived tool use.

What SB‑1047 Does (per the thread)

• Applies to “covered models” defined by a compute threshold (~10²⁶ FLOPs) or comparable benchmark performance to state‑of‑the‑art 2024 models.
• Creates a Frontier Model Division to collect reports, issue guidelines, and assess fees; can recommend shutdowns and civil penalties for violations.
• Requires training‑time safety determinations, incident reporting, “full shutdown” capability, and certification under penalty of perjury for frontier models.
• Defines catastrophic harms (e.g., WMD use, massive cyber damage, high‑dollar autonomous crimes).

Impact on Open Source and Startups / Regulatory Capture

• Many fear this will entrench big incumbents (OpenAI, major cloud providers) and lock out smaller players who lack legal resources.
• Some see this as classic regulatory capture: a few large firms pushing rules that hurt open‑source and new entrants while being cheap to lobby for.
• Others counter that the bill is targeted at only the very largest/most capable models and explicitly contemplates incentives and guidance for non‑hazardous open‑source work.

Arguments That the Bill Is Reasonable and Narrow

• Supportive voices say it is not a licensing regime and does not impose strict liability. It asks only for “reasonable measures” and reporting for models capable of catastrophic harm.
• They emphasize that current open models likely do not meet the threshold and that the scope is far narrower than, for example, the EU AI Act.

AI Safety, Risk Perception, and Regulation Target

• One camp sees advanced AI as potentially existential or at least massively destabilizing; they view preemptive regulation as analogous to safety rules in other high‑risk domains.
• Another camp sees frontier models as “just math” or “autocomplete,” arguing harms are speculative and that regulation is really about speech control and corporate profit.
• There’s a strong split over whether to regulate development (frontier training) versus use (specific applications like healthcare, weapons, deepfakes). Critics say policing development is infeasible and overbroad; supporters argue that controlling only usage misses export, military, and global risks.

Technical Thresholds and Feasibility Concerns

• Commenters debate whether the 10²⁶‑FLOP line is “absurd,” already near Llama‑class models, or quickly surpassed due to algorithmic and hardware improvements.
• Some note definitional issues (ints vs floats, precision, benchmark gaming) and clauses that seem impossible to satisfy (e.g., preventing any derivative model from causing specified harms).
• Others say having to test, maintain a shutdown mechanism, and report on hazardous capabilities is reasonable for actors with that level of compute.

State, Global, and Enforcement Considerations

• Some foresee AI companies leaving California for states with lighter regulation and cheaper power/real estate (e.g., Texas, Washington, others), while noting talent and research ecosystems matter.
• Comparisons are made to past crypto export controls and DMCA‑style rules; skeptics question the practicality of controlling anonymous open‑source releases.
• A few warn that over‑regulating in one jurisdiction will just push cutting‑edge work, including potentially dangerous work, to other countries with fewer safeguards.