Yesterday we were amused to see this post on Reddit's sysadmin forum:
Our architecture lead Nick Craver looked into this and gave a great answer, including that about 29% of the previous day's Stack Overflow traffic was to the home page. From his perspective as a system administrator, that's exactly the right analysis. Nick's team has to ensure the site renders reliably and quickly, so counting the number of requests make sense. However, as a data scientist I'm more interested in how users view the site, and the vast majority of our traffic comes from automated traffic, including search engines and scrapers. When real people view Stack Overflow, how often are they visiting just questions? Who does and doesn't visit the homepage, or other parts of the site?
What pages do people view?
If we look at yesterday's traffic (2017-03-08), including automated traffic, here are the most common types of page views.
(Note that this is only counting pageviews, not asynchronous requests that load after a page or internal API calls; sysadmins do care about the latter). Each of these represents one route: typically one possible type of page. The most common include:
- Questions/Show: question pages like this one, the kind of page most people associate with Stack Overflow
- Home/Index: our homepage
- UsersShow/Show: visits to a user’s profile (here’s mine)
- Questions/ListByTag: Pages with the most recent questions from a tag, like this one
It does look like a lot of people are visiting the homepage, nearly as many as are visiting questions. However, that's because we're counting automated traffic! As a next step, I'll take a few simple measures to remove automated traffic and include only people we think are real users. These measures won't catch a savvy scraper who "doesn't want to be caught", but they easily filter out your "everyday" bot.
Almost 95% of our non-automated traffic is to question views, with only a small percentage to the home page or other views. This isn't surprising (it's the motivation behind the Reddit post), but it's a good insight into typical visitor behavior.
How do registered visitors use the site?
When I tell a software developer I work at Stack Overflow, they almost always recognize it but often admit that they don't have an account. That's true of most of our users; only about 15% of our daily traffic is logged in when they visit. Most developers visit Stack Overflow questions from a Google search as they solve their problems, then move on with their work. We owe a lot to the community that's registered on the site, however, because they're responsible for our terrific set of questions and answers. As you might expect, this part of our community visits the site in a different way.
Within unregistered traffic, the percentage of traffic that goes directly to questions is up to 97%. The Reddit question was right! But a typical registered visitor, who might have answered some questions, does spend some time on the home page, as well as visiting user profiles and tag lists of recent questions.
Power users
While we're grateful to everyone who contributes to the site, some within the community ("power users") contribute an enormous amount of knowledge We usually represent this in terms of reputation, gained when content you've contributed is upvoted. Two thresholds we use for user privileges are established users, who have at least 1,000 reputation, and trusted users, who have at least 20,000 reputation. To give a sense of these levels: among users who visit in a typical week, the median user with less than 1,000 rep has answered about 3 Stack Overflow questions, the median established user has answered about 50, and the median trusted user has answered about 750 (!). How do power users visit the site?
Trusted users (20,000+ reputation) spend about half their time viewing actual questions, and most of the other half visiting the homepage, question lists, or user pages. The home page offers suggestions of questions that are both new and likely to be relevant to you, so this makes sense. The ListByTag route is also relevant as an insight into how "power answerers" use the site; many stick to a particular tag that they're familiar with, and watch for brand new questions they can answer. We're happy that we make the lives of developers easier, even if they're just getting a quick solution after a search. But we'd also encourage you to join the world's largest developer community, whether to ask and answer questions, get your next job, or build your online presence with a Developer Story. In any case, next time you solve your problem through Stack Overflow, remember the hundreds of thousands of users who regularly ask, answer, edit, and moderate the site to make it all possible.