Thanksgiving 2023 security incident
Cloudflare’s detailed postmortem on its Thanksgiving 2023 breach — traced to credentials stolen in a prior Okta compromise that weren’t fully rotated — has prompted both praise and skepticism. Commenters highlight the company’s aggressive response (reimaging thousands of systems, rotating ~5,000 secrets, and even replacing hardware in a new data center) as a model incident response, while others question residual risks from exposed source code and Jira/Confluence data, the continued reliance on Okta, and gaps between “zero trust” marketing and actual access controls.
Incident Response and Remediation
- Many are impressed by the “nuclear” response: rotating ~5,000 credentials, triaging ~4,900 systems, reimaging/rebooting the global fleet, and physically segmenting environments.
- Several say this goes beyond what most large vendors do after breaches and makes renewals/sales easier.
- Others argue this level of response should be table stakes, not exceptional, and note that past major breaches at other firms had little long‑term business impact.
- Some recall earlier incidents (like parser bugs) and question whether Cloudflare has always been as transparent as it now presents itself.
Okta, Credentials, and Zero Trust
- The intrusion stems from credentials taken in an earlier Okta support compromise that Cloudflare failed to rotate fully.
- Debate over blame: some see “punching down” at Okta to deflect from Cloudflare’s rotation miss; others say Okta’s repeated failures warrant harsh criticism.
- Questioning of “zero trust” marketing: attackers leveraged a single bearer/service token and service accounts to reach Atlassian; critics say true zero trust would further constrain this.
- There is discussion about why “unused” or poorly understood credentials weren’t simply revoked or deleted, versus the operational fear of breaking unknown dependencies.
Scope of Access and Impact
- Attackers accessed a small fraction of Jira tickets, wiki pages, and repos, apparently focused on architecture and access patterns rather than customer data.
- Some note Confluence/Jira search may leak secrets without opening pages, making true impact harder to measure.
- Skeptics argue that any exposure of internal source and bug reports is serious and permanently increases risk; others emphasize that only limited, internal systems were touched.
Tooling Choices and Third‑Party Integrations
- Surprise that a large infra provider uses Bitbucket; defenders cite Atlassian integration and cost.
- Concern that a Smartsheet service account with Jira admin rights and ScriptRunner allowed Sliver C2 installation, highlighting risks of powerful third‑party integrations.
Hardware Replacement and Waste
- Returning and replacing equipment at the new Brazil data center is praised as extremely thorough by some and condemned as wasteful by others.
- Counterpoint: for critical network gear with potentially untrustworthy firmware, replacement is seen as the only way to be confident.
Trust, Transparency, and Compliance
- Many say detailed write‑ups and visible overreaction increase their trust in Cloudflare.
- Others call the blog post “advertising,” arguing it emphasizes remediation while downplaying how the situation arose.
- Side debate over PCI/SOC: some claim external forensics are compliance‑driven, others respond that PCI is mostly checklist‑based and not the main driver here.
Side Discussion: Personal Use of Work Devices
- Long tangent about Okta device management and whether to keep any personal accounts/passwords on corporate laptops.
- Opinions range from “absolutely nothing personal on work devices” to “light personal use is inevitable and policies should assume endpoints are compromised.”