Normally Kevin does these updates, but I wanted to take an opportunity to talk about how Documentation is doing from my perspective as a community manager. For the last 3 years or so, building communities of specialists has been my specialty. In my experience, communities on Stack Exchange have similar lifecycles:
A new topic is introduced, a bunch of early adopters get excited about it and the first few days feel like a runaway train hurtling up and to the right.
Reality begins to set in.The community is losing momentum. Folks wander off to do something else. Excitement dissipates and more people start looking for the exits.
One of two things happen next: either the community slows to a virtual halt, or a handful of dedicated users get down to the real work of building a community.
As Shog9 showed, we passed the first stage months ago.
What's happened since launch?
Most of the work in a product's beta period aims to build the framework of community. For Stack Overflow Q&A, that framework consisted of questions, answers, voting, reputation, privileges (especially editing), curation and lightweight commenting. Those systems working in concert enable "a bunch of amateurs muddling along trying to do things together". Since Q&A is a resource most programmers use daily, we sometimes forget the struggle required to build the underlying software.
2016 has been about discovering and building the systems needed for user-contributed, example-first, Stack Overflow-integrated Documentation. Much of that work involves creating and testing hypotheses with actual users. It isn't quite science. But the critical element has always been your feedback — expressed explicitly on meta and implicitly via your actions.
While reputation seemed to work in the confined space of a private beta, it was abundantly clear at launch the reputation system didn't work. So we changed it. Twice. There's every chance we'll tweak it again, but the philosophy of rewarding work on Documentation roughly at the same rate as Q&A will remain. Currently, the median Documentation contributor has earned 7 points of reputation (up from 6 in September) compared to 10 points for the median outside of Docs. Since launch, less than 1% of Stack Overflow reputation has come from Documentation, which roughly coincides with activity. It's not a perfect system, but it seems reasonably consistent with the rest of Stack Overflow at the moment.
Unlike questions and answers, most contributions to Documentation are subject to review. Initially, we hoped that people would watch the tags they were interested in and that reviews would happen organically. We were wrong. After adding a global review queue (with 2.3k items!), we (re)discovered that some people just love to make tasks go away. So we added audits and review bans. It also seemed reasonable to give experienced users more influence on the queue, so we gave reviews from high-reputation users and silver tag badge holders more weight.
The public launch revealed bugs. Lots and lots of bugs. In fact the feedback has been a bit overwhelming. Many of the changes made (or planned) since beta have started in meta: reputation changes, review queue, rejection reasons, plagiarism warnings, improvement request refactoring, and a discussion area for example. We've done a lot in 2016 considering our team consists of 3 developers, a designer, and me, the resident cheerleader. It's inevitable that some important feature requests and bugs haven't been implemented. We'll be onboarding a dedicated product manager soon, who will help us prioritize future work. Kevin has done a solid job in the role, but he's been stretched too thin for too long.
What's next?
We aren't quite ready to roll out the final major piece of the system, which is a method of discussing changes and keeping track of those decisions. There has to be some way for people to talk about their policies and practices, and so we put a fair amount of effort in the R&D stage of the new "discussion" tab. I have interviewed a number of people who were given an early view of the static prototype. Those user tests revealed decisions we made that were less logical than we thought. This sort of feature is easy to get wrong, but I think we are on the right track. As with everything else, the real test will come when the feature is publicly available, so we’ll be ready to iterate on that as well.
Just this week, the executive team gave us a key metric for measuring success in the coming year: user growth. Obviously there’s a business reason for that goal, but it serendipitously benefits the quality of Documentation as well. As Eric Raymond notes, “Given enough eyeballs, all bugs are shallow.” Similarly, the more people who use Documentation to understand their code, learn a new feature or even answer questions on Stack Overflow, the more reliable the content will become.
There are a number of levers we can pull to give Documentation more pageviews. But we've been holding off on doing those things until the system for participating is solid. In large part it means handling feedback about particular Topics and Examples. Sometimes it's enough to say "this is confusing". More often, a user will need room to explain exactly what is confusing and talk about what changes might improve the situation. Meanwhile, it's been difficult for people who understand the system to guide those who don't. Meta is no substitute for comments right next to the content itself, so adding more eyeballs won't help until that piece of the puzzle is in place.
It's no secret we've been looking to Wikipedia for inspiration. Benjamin Mako Hill studied eight online encyclopedias and concluded:
Wikipedia offered low transaction costs to participation, and it de-emphasized the social ownership of content. Editing Wikipedia is easy, and instant, and virtually commitment-free. “You can come along and do a drive-by edit and never make a contribution again,” Hill pointed out. And the fact that it’s difficult to tell who wrote an article, or who edited it — rather than discouraging contribution, as you might assume — actually encouraged contributions, Hill found. “Low textual ownership resulted in more collaboration,” he put it.
We've resisted adding restrictions on new-user participation because Stack Overflow is built on "little tiny pieces of information". Even to this day, many good answers are given by people who don't care about Stack Overflow, but do care about some language feature, correcting misinformation or whatnot. The long-term quality of Documentation will depend on corrections, additions and clarifications from the people trying to use it.
With the software skeleton nearing completion, the real work of building a community is just beginning. On behalf of everyone on the Docs team, thank you for all of your help and advice in 2016; onward to 2017!
