Production incident · Diagnosis and remediation
DDoS incident caused by an exposed origin
Production incident where the protection layer could be bypassed
A client’s production service was under a DDoS attack. It sat behind Cloudflare, but the origin server still accepted direct connections from the internet.
Problem
The application ran in a Docker container with published ports and no reverse proxy in front of it. Docker adds its own firewall rules for published ports, so UFW didn’t filter them, and anyone who found the origin IP could skip Cloudflare entirely.
Result
With the origin closed, the traffic went through Cloudflare again and the edge rules could do their job. Legitimate users got stable access back. I don’t have traffic or attack figures from the incident, so there are none here.
Decisions and implementation
- Tightened the Cloudflare WAF and rate limits first. The traffic reaching the application didn’t drop, and Cloudflare didn’t show the load that was flooding the container logs. That mismatch pointed to traffic that wasn’t going through Cloudflare at all.
- Checked the origin in Censys and Shodan, which confirmed it was exposed, and compared the Cloudflare data with the origin logs.
- Changed and isolated the origin IP and limited the ways the server could be reached.
- Tuned the WAF and rate limiting once the direct path was closed, trying not to get in the way of normal visitors. I also used Sefinek’s community-maintained Cloudflare-WAF-Expressions ruleset.
- Cached database queries, views and routes in the application, so each request did less work.
Technologies used
- Cloudflare
- WAF
- Rate limiting
- Docker
- Linux
- Origin isolation
- Application caching
- Log analysis
Notes from this project
Technical notes related to this project:
Related competencies
Related technology areas:
Related work history: experience.
All case studiesNext case study: Device provisioning platform