On Monday afternoon, we unceremoniously closed down a Stack Exchange site: Artificial Intelligence.
Not that many would have noticed. It had an laggardly 83 questions in its 12 days of existence. It wasn't so much the lack of questions that was of concern — a site can stay in beta as long as it takes — but the conspicuous lack of _expert-level _questions. This was also the emerging opinion amongst the users:
I committed to this proposal some time ago, hoping that this might become a site for researchers or knowledgeable academics asking serious technical questions about artificial intelligence here. It seems I was dearly mistaken ... Most of the questions are those asked by the merely curious.
70-80% of the questions didn't run much deeper than "When will we have intelligent computers?" and "What is your favorite AI blog?"
I can understand the curiosity. As a computer enthusiast, I am somewhat intrigued by artificial intelligence. But I couldn't even begin to ask a question suitable for a knowledgeable researcher. I'd be one of the merely curious… as were most users on the site.
AI's problems began almost immediately when users started asking the first questions:
It has long been established that no question is too entry-level nor too basic. Everyone is welcome. But, in these earliest days, we are DESIGNING a site for experts. To attract experts, you need a site where people are asking very interesting and challenging questions, not the basic questions found on every other Q&A site. Remember, the pro sites WILL attract the enthusiasts, but not the other way around! The earliest questions on a site will set the tone and topic of the site for a long time.
The AI site conspicuously lacked that "tone and topic" from day one, so it had nowhere to go and was closed down.
Lessons for Area 51
The purpose of Area 51 is to prove that a site has critical mass before it launches. That didn't work here. The followers and committers accumulated very slowly over six months, so there was a low response rate from the committers to join the private beta. We may need to consider something like aging commit votes so older proposals don’t simply accrete votes over a long period of time. We need to establish that a proposal has sufficient momentum (an escape velocity) before it creates a site.
It is difficult to say if many experts were part of the AI proposal; I don’t think we ever had them. But Area 51 has no way to measure if the private beta will include a sufficient number of qualified practitioners, so we need to make adjustments. Here are some changes we adopted from this experience:
A site can stay in private beta. Typically, a site is ready to launch after their prerequisite seven days in private beta. But if the early site hasn’t worked out the kinks of its definition or audience, we can work with the community before launching a half-cocked site to the public. If the problems are impossible to reconcile, the site can always be closed to try again. As we improve the process of Area 51, this should be exceedingly rare, but AI has become the new baseline for “failed to make it out of private beta.”
Establish a site's expertise early. New users, anxious to jump start their communities, inevitably start asking uninspired questions that have all been asked 100 times before on every phpBB forum. You've seen them: "What is your favorite...", "What is the best...", "What is the definition of..." Unfortunately, these idle questions can fill the front page in the opening days, and left unchecked will permanently color the tone of the site. Your front page is your billboard, it defines your target audience.
Watch for and discourage pedestrian questions early in the beta. Certainly, questions of all levels are welcome… but not in the earliest, most formative days of the site. These questions may someday become wildly popular, so politely --- very politely --- invite those users to ask their questions again after the expertise of the site has been established. Top blogs, best books, buying recommendations: those are not the hallmarks of expertise. They're the seeds of the merely curious. A site filled with these sorts of idle, pedestrian questions will never attract the core of experts it needs to survive.