About a week ago, I started running a little survey asking Wikimedians how we should approach target-setting for the next five years.
I did it because next month Wikimedia will finalize the targets that’ll guide our work for the next five years, and I wanted to gather some quick feedback on the thinking that’s been done on that, to date. The survey’s close to wrapping up now, and the results thus far are terrific: there appears to be good consensus on what we want to measure, as well as on our general approach.
More detail below! But first, some general background.
In July 2009, the Wikimedia Foundation kicked off a massive strategy development project, which is starting to wrap up now. [1] The one major set of decisions that remains to be finalized is how we will measure progress towards our goals.
The draft goals, measures of success and targets that have been developed via the strategy project are here. They were created over the past several months by Wikimedia community members, Bridgespan staff, and Wikimedia Foundation staff (thank you all) – and in my opinion, they’re pretty good. They focus on what’s important, and they do a reasonably good job of figuring out how to measure things that don’t always lend themselves to easy measurement.
Before finalizing the targets and taking them to the Wikimedia Board of Trustees for approval, I wanted to gather some additional input, so I hacked together a quick, imperfect little survey. (You can read it –and fill it out if you want– here.) The purpose of this post is just to share the results — I will probably write more about the targets themselves later.
First some methodology: I made the survey in Google Docs, and sent identical versions to i) the Wikimedia Board, ii) the Wikimedia staff, and iii) the “foundation-l” mailing list (a public list on which anyone can talk about the Wikimedia Foundation and Wikimedia projects), the Wikimedia Foundation Advisory Board list, and the “internal-l” mailing list (a private list intended for Wikimedia chapters representatives and Wikimedia Foundation board and staff). Then –for the purposes of this post– I aggregated together all three sets of results, which total about 120 individual responses thus far.
If I’d been more serious I’d have used LimeSurvey, which is a better survey tool than Google Docs — but this is really just meant to be a structured solicitation of input, rather than a proper quantitative study. For one thing, the “community” results reflect only a tiny fraction of active editors — those who read English, who are on Wikimedia’s mailing lists or are connected with people who are, and who self-selected to answer the survey. So, please resist the temptation to over-interpret whatever numbers I’ve given here.
In general, I was happy to find that the survey surfaced lots of consensus. A comfortable majority agrees with all of the following:
- Wikimedia’s goals should be “ambitious but possible.” (Other less-popular options were: “definitely attainable, but not necessarily easily,” “audacious and probably not attainable, but inspiring,” and “fairly easily attainable.”)
- We agree that the purpose of setting goals is “to create a shared understanding and alignment about what we’re trying to do, publicly and with everyone.” (Other options: “to create an audacious target that everyone can get excited about and rally behind,” and “to create accountability.”)
- In setting goals, we believe “perfection is the enemy of the good: I would rather see us using imperfect measures than no measures at all.” (About 15% of respondents felt otherwise, believing that “imperfect measures are a waste of time and energy.”)
- The Wikimedia Foundation’s goals should be dependent on efforts by both the Wikimedia Foundation and the Wikimedia community, not by the Foundation alone. (18% of respondents felt otherwise, that the targets should be “entirely within the control of the Wikimedia Foundation to influence.”)
- If we exceed our goals, practically everyone will be “thrilled.” (About five percent of respondents felt otherwise, saying that they would be “disappointed: that would tell me our goals weren’t sufficiently challenging.”
- If we fail to meet our goals, about three quarters of respondents will feel “fine, because goals are meant to aspire/align: if we do good work but don’t meet them, that’s okay.” Interestingly, this is one of the few areas of the survey where there was a real division between the staff of the Wikimedia Foundation and other respondents. Only 17% of staff agreed they’d be okay with missing our targets. I think this is probably good, because it suggests that the staff feel a high sense of personal responsibility for their work.
- Almost everyone agrees that “goal-setting for the Wikimedia Foundation is difficult. We should set goals now, but many measures and targets will be provisional, and we’ll definitely need to REFINE them over the next five years, possibly radically.” (Runner-up response: “we can set good goals, measures and targets now, and we should NOT need to change them much during the next five years.” And a very small number felt that we should refrain from setting targets for “things we’re still uncertain about,” and instead restrict ourselves to areas that are “straightforward.”)
- The global unique visitors target is felt by most to be “attainable if the staff and community work together to achieve it.” (About 20% of respondents felt the target might be “even happen without any particular intervention.”)
I wanted to get a sense of what measures people felt were most important. They’re below, in descending order of importance. (The number is the percentage of total respondents who characterized the measure as either “critical” or “important.” Other options were “somewhat important,” “not important,” and “don’t know/not sure.”)
It’s probably worth noting that consensus among community members, the board and the staff was very high. For more than half the measures, the percentage of respondents rating the measure as “important” or “critical” varied by less than 10% among the different groups, and for the remainder, it varied by less than 20%.
-
Measure Avg Retention of active editors 84 Number of active editors 83 Site performance in different geographies 80 Demographics of active editors 80 Uptime of all key services 78 Financial stability 74 Global unique visitors 66 Secure off-site copies 65 Number of articles/objects/resources 65 Regular snapshots/archives 60 Thriving research community 54 Offline reach 53 Reader-submitted quality assessments 41 Expert article assessments 40 Community-originated gadgets/tools/extensions 22
The survey’s still accepting input — if you’re interested you’ve got until roughly 7PM UTC, Wednesday August 18, to fill it out.
————————————————————————————–
[1]
I launched the Wikimedia strategy project at the request of the Wikimedia Foundation Board of Trustees, and it was led by Eugene Eric Kim of Blue Oxen Associates, a consulting firm with a special focus on enabling collaborative process. Eugene worked with Philippe Beaudette, a longtime Wikipedian and online facilitator for the project, and The Bridgespan Group, a non-profit strategy consulting firm that provided data and analysis for us. The premise of the project was that the Wikimedia movement had achieved amazing things (the number five most-used site in the world! 375 million visitors monthly!), and it was now time to reflect on where we were making good progress towards fulfilling the mission, and where we weren’t. With the goal of course-correcting where we weren’t doing well.
To come up with a good plan, we wanted to stay true to our core and central premise: that open, mass collaboration is the most effective method for achieving high-quality decisionmaking. So, we designed the process to be transparent, participatory and collaborative. So, during the course of the project, more than a thousand volunteers worked together in 50+ languages — in teams and as individuals, mostly in public on the strategy wiki, but supplemented by IRC meetings, Skype calls, e-mail exchanges, and face-to-face conversations (e.g., meetings were held in Berlin, Paris, Buenos Aires, San Francisco, Boston and Gdansk).
The project’s now entering its final phase, and you can see the near-final results here on the strategy wiki. What remains to be done is the finalization of the measures of success, which will happen over the next six or so weeks. At that point, there will be some final wordsmithing, and the result will be brought to the Wikimedia Board of Trustees for approval.
I will probably write about the strategy project at a later date, because it is super-interesting. (Meanwhile, if you’re interested, you can read a little about it here in a story that Noam Cohen wrote from Wikimania 2010 in Gdansk.)
August 17 is Tuesday, Wednesday is 18th.
You’re right Kozuch, thanks. It’s Wednesday the 18th that I meant; I’ve corrected the post.
Dear Sue,
it is great that you publish poll results. I am really happy to see this rather than nothing, but as someone said – Wikimedia cant be glued together by tape, and if you were using Google Docs for an in my opinion TERRIBLY IMPORTANT poll that can influence future of Wikimedia for several years (okay we can refine goals but still), this is a real sign of not being professional at all…
I am an Admin over at the Strategy planning wiki, and I was very said seeing a wiki (=MediaWiki) to be the tool that should produce strategic plan. The article “Wiki” is a top visited article on English Wikipedia, that means almost no-one from general public (people who do not edit Wikimedia and therefore a prioriti is to get them) means what a wiki is and how to contribute to it…
Dont you want to run a decent questionare at http://survey.wikimedia.org/ ??? We still have some time. Much better would be running such questionaries repeatedly as a support for the Strategy planning… wikies usually end up as a collection of garbage, if you look at the strategy wiki it is not the “door” to Wikimedia as it whould be.
What about http://getsatisfaction.com/wikipedia ??? Seems to be dead. What about IdeaTorrent? There need to be tools that easily collect input from general public, and a wiki is not such a tool… if you understand what I mean.
Hi Kozuch. I’m comfortable with the Google questionnaire. Bear in mind that it’s not the only tool we’ve used to develop the goals — it’s just a final sanity-check, and one more way for people to offer input.
I don’t agree that wikis usually end up ‘as a collection of garbage’ — the Wikimedia projects in general suggest that’s not true, and I think the strategy wiki was really well facilitated. There is lots of good work there.
You dont understand me. I am speaking about demographics of strategy wiki contributors…
Ah, I’m sorry, I understand now. And yes, you have a very good point.
When we chose a wiki as the main staging ground for the project, we knew it would have the effect of disenfranchising some people who might otherwise participate. On balance, we decided that the public and open nature of the wiki was critical to the project’s success, and that we would take other measures to bring in voices that a lack of wiki-facility would exclude.
And we did some of that — for example, we conducted F2F and phone interviews with some advisory board members, and some experts from organizations like eBay India, eBay China, Greenpeace, Habitat for Humanity, etc.
But in general, you’re right. We knew it would be a problem, and we took some steps to mitigate it. But working on a wiki definitely cost us some contributors.
A side note: I have really got to do something about how comments display here. This margin is ridiculous.
(I changed to a different theme, so that comments can nest more than three levels deep without being unreadable. I will probably still futz around with it a little over the next few weeks.)
Dear Sue, I am very happy we found similar language. I am one of the Wikimedia editors who is constantly balancing on the edge of leaving the projects… that is why I do not have that many content edits (=I do not create a lot of content on Wikipedia for example) but I constantly try to work on the whole (eco-)system to get better, because that is the only way how editors (selfishly including myself) can be kept from leaving. I try to do a lot at the “backstage projects” like Meta, Strategy, Usability, Outreach and others, mostly linking things together so that they are easier to find and hopefully bringing some new ideas to work.
It is terribly painful for me to see great proposals burried under all the stuff at the Strategy wiki. It is terribly painful for me to see that bugzilla/mediazilla has a very limited scope (read it is not very accessible to newbies) within Wikimedia. I hope things will get better with all the new hiring, hopefully more staff will have time to really support the community and especially listen to what the community exactly wants and needs.
After rough four years with the projects now I am still only an editor at the English Wikipedia and even in todays admin drought I am not aspiring to run for an RfA – simply becuase it does not give me more power besides content creation – and that is what I do not (generally) do. I am pointlessly looking at how to step up a bit in my “volunteering career”. Simply link building is nice, but frustrating after some time… the problem is the environment is so extremely diverse, that one is often failing to find his/her own niche to work on (despite this contradicts itself)… and what more, those niches need to be connected once together to deliver the power/work/features that readers/editors/admins want. If you can think about this a bit I would appreciate it.
I simply am asking on how can I continue to help in the backstage projects if I see (in my opinion only of course) no or very little development in this field and I do not have power enough to change certain things that bother me the most. Of course I could apply for a staff position in foundation, but that is not my intention at least for a while yet (mostly for geographical reasons). I am missing some opportunities for lets say power-editors like me and am asking you to offer some.
Of course, the opportunities are there but they are extremely diverse – and in my point of view either extremely easy (content creation in my opinion) or extremely hard (coding MediaWiki). I am missing the “medium” option here. I try to get involved with Bugzilla, but this is rather “medium hard” right now…
When speaking about Wikimedia goals in general, hopefully after the Strategic plan is ready, it will break up in lots of goals and those in lots of tasks of various importance and difficulty. This could faciliate better distribution of workload to the volunteers, where everyone could find his/her own.
Thanks for reading :-).
Dear Sue, will you reply to some of my points and possibly take some action? Retention of active editors is a critical point as we see from the table above…
Why is telecommuting not a measure of success? Are there any measures of how well the Foundation and its employees are setting examples with sustainable practices? Will the Foundation measure how well it and its employees act in direct support of positive social change in the locales they are situated? Are sustainable business practices worth measuring?
Thanks for sharing the survey results.
I feel very positive about the order in which the results came out. The only one I was slightly saddened to see was the low status given to gadgets. I have some gadgets installed that make my life on Wikipedia so much easier and really help my productivity.
However, I realise that gadgets were up against stiff competition in this list of priorities. Also I think that people who produce gadgets are a particular demographic who will likely continue to program wonderful things without much outside encouragement. If the forecasts about the increase in tech staff turn out to come true, I would still imagine their role not to be to produce gadgets; although it would be nice if there were a tech staffer who had a role in liaising with our freelance software tweakers.
I was a little surprised by the results on gadgets too, Bodnotbod. I think to a certain extent gadgets suffered because, as you say, they appeal to a particular subset of the community — people who are particularly engaged, say, and technically proficient, and understanding about how the Wikimedia projects work. (Although I’d have assumed those would be the folks who would have self-selected to respond to the survey in the first place.)
I get that “number of community-produced gadgets” isn’t the kind of measure we’d necessarily trumpet from the rooftops, because it wouldn’t be hugely understandable/motivating for donors and readers… but I do believe it’s an important measure, and it speaks to community health as well as supporting quality, IMO.
In addition to what Sue said, one of the strategic priorities was to encourage innovation… so I was surprised to see so little interest in that type of innovation.
Could that have something to do with the way the items are written? “Supporting innovation” sounds a lot better than “gadgets/tools/extensions” to me.
I tried to convert these all to grammatical verb phrases and add some balance to areas which have been overlooked at http://lists.wikimedia.org/pipermail/wikitech-l/2010-September/049268.html Please have a look.
By the way — another somewhat surprising result, to me, was “site uptime” — highlighted in red in the table. It’s highlighted because it was the measure that received the least consensus. (The range of responses was largest.) I hesitate to over-interpret that, for the reasons I describe above. But it’s worth noting. To me, good uptime is the price of admission: the Wikimedia sites are the fifth-most-popular on the internet, and our readers deserve to have the service consistently available.
Agree 100%.