Server

background December 13, 2009

Blog Outage – Backup Policies

On Friday, the server which hosts this blog suffered catastrophic data loss. Fortunately, the blog server is at a different datacenter entirely than PEAK, which hosts all the Trilogy sites. It’s a long story, and I’ll document it in more detail elsewhere, but the short version is this: This particular host’s (again, not PEAK) backup…
Avatar for Jeff Atwood
Co-Founder (Former)
company December 5, 2009

Database Server Upgrade: Outage Saturday

There will be an outage today from 5 pm – 7 pm PST as we upgrade the database server. If you’re curious, we’re going from 2.5 GHz to 3.33 GHz CPUs, and using the old CPUs to build a second, backup database server. Our last upgrade was in July when we went from 24 GB…
Avatar for Jeff Atwood
Co-Founder (Former)
company October 16, 2009

Stack Overflow Outage

Our apologies for the all-site outage today. According to our Pingdom monitors, we were down from 7:18 PM PST to 9:43 PM PST. There goes our vaunted envy-of-the-industry three nines uptime guarantee! Apparently there was a router meltdown at our ISP, Peak Internet. They promised pictures of the (literally?) melted router via an update on…
Avatar for Jeff Atwood
Co-Founder (Former)
background September 11, 2009

New DNS Provider

Our domain name registrar is GoDaddy. We’ve had a lot of problems with GoDaddy’s handling of DNS, where DNS entries will suddenly appear and disappear at random. Often, changing a completely unrelated DNS record would result in other DNS entries going missing for hours. Extremely frustrating. As a result of many, many bad experiences, over…
Avatar for Jeff Atwood
Co-Founder (Former)
company September 4, 2009

Load Balancing Stack Overflow

Starting right now, we will be load balancing the Stack Overflow servers — going from one web tier server, to two. This means you may end up on a different server depending on what HAProxy decides the hash of your IP address is. This shouldn’t cause any problems, but … If you notice anything unusual,…
Avatar for Jeff Atwood
Co-Founder (Former)
background July 17, 2009

High Scalability Blog on Stack Overflow

It was quite an honor to see that the High Scalability Blog posted an entry on Stack Overflow! We referred to the HSB, and its exhaustively detailed information about how other websites handle scaling, many times during the course of Stack Overflow development. And I’ve cited it myself when researching what we think is the…
Avatar for Jeff Atwood
Co-Founder (Former)
company July 5, 2009

Database Server Upgrade — 48 GB

We are currently upgrading the database server to 48 GB of memory, which also means we have to upgrade the operating system, too. Should be done in about an hour. OK, this is complete. Our database server not only has 48 GB of memory installed, but has access to all of that memory. Finally. And…
Avatar for Jeff Atwood
Co-Founder (Former)
background June 16, 2009

The Perfect Web Spider Storm

We noticed something unusual on our Cacti graphs today. Can you spot it? Yes! The light gray of the graph background does seem a few shades lighter than normal! I see it too! No, no, of course I’m talking about that massive traffic spike from 06:00 to 15:00 PST (server time). In the words of…
Avatar for Jeff Atwood
Co-Founder (Former)
company April 1, 2009

Tuesday Outage: It”s RAID-tastic!

We had a brief outage early Tuesday morning from 3 AM – 5 AM PST, because the database server was doing this: Oh noes! Not … !!! CRITICAL ERROR: Memory retention failure, unflushed cache lost !!! There are six exclamation points so you know it’s serious. Also, you have to press ENTER. Because it’s a…
Avatar for Jeff Atwood
Co-Founder (Former)
company February 20, 2009

Our Backup Strategy – Inexpensive NAS

As part of our new backup strategy, I decided to invest in an inexpensive Network Attached Storage (NAS) unit — the QNAP TS-409U. The TS-409U has the following specs: 1u rack mountable 4 SATA drive bays 500 MHz embedded CPU, 8 MB flash RAM, 512 MB RAM, Linux based Supports RAID 0, 1, 5, 6…
Avatar for Jeff Atwood
Co-Founder (Former)
company February 8, 2009

New Datacenter Migration

We just finished migrating our new servers over to the new datacenter at PEAK Internet. The installation was aided greatly by Stack Overflow team member Geoff Dalgas, who happens to live about a mile from PEAK in Corvallis, Oregon. He visited the facility before the servers arrived, and also helped with their installation. Having Geoff…
Avatar for Jeff Atwood
Co-Founder (Former)