As I'm sure many have noticed, over the past few weeks the site has run slow and has had random periods of downtime lasting between one and five minutes. This was due to a combination of factors, including:
- Kinmunity suffered a Distributed Denial of Service attack, originating from a Chinese botnet. As an emergency mitigation measure, we blocked Chinese hosts from connecting to ikari (our primary server).
- We have had a surge in guest (unregistered user) traffic to certain articles and pages on Kinmunity.
- ikari has gone from hosting only one, to hosting four distinct web properties.
- ikari's backup system now needs to backup each web property (files and database), causing increased performance use during backup generation.
- Kinmunity's cache system was storing a lot of things in RAM rather than on disk; we have much more disk space than we do RAM.
- We are now limiting the use of memcache. It's still used for some things, but not nearly as much data will be stored in it.
- We are now caching a lot of (mostly) static content to our disk. When guests (unregistered users) load such content, a lot less resources are used to display it to them.
- We have implemented anti-DDoS and anti-DoS measures at the software level; our server host does a great job mitigating network-level attacks, but not application-level. That gap is now closed.
- Perhaps the most important change; we have changed our multi-processing-module in Apache; causing our web server to use much much fewer resources in regular operation.