John Oliver said:
Nothing will happen within the next year. If you can't build a simple scalable system that can take up to 100,000 hits per second to run on substandard hardware then something's gone hideously wrong.
Come on you guys are supposed to be talented.
We never set out to build a scalable system, or any particular type of system at all. The site started as a few pages we threw together in our free time, with no real consideration given to growth, and over the years we have grafted more and more on to that poorly designed core.
Now that we all have careers, free time has dramatically reduced, which means we aren't in a position to start from scratch ourselves - else we would have done so two years ago. Employing others to do the work for us required money that we didn't have, so we endeavoured to raise some, and you've no doubt seen that most of our efforts have had that flavour over the past two years.
We're finally in a position where we can start considering options that will make a real difference to the site, and should come to fruition late next year.
jm01 said:
This site isn't that bad compared to others that I've seen in terms of activity. Are the servers' services seperated (ie. dedicated server for mysql and web)?
What are the specs like?
There are dedicated servers for the www and community subdomains as well as one for mysql. The problem isn't so much the adequacy of the hardware as the way in which the servers are configured - which is very sub-optimal.
Reworking that configuration by aiming for an optimal solution (rather than the solution which requires the least effort) is part of the plan to fix the performance issues.