Documentation metrics: How do you prove you’re worth it?
January 15th, 2010 16 Comments
You know how sometimes you read something and it niggles away at you and you can’t quite get it out of your system? This happened to me with a blog post by Ivan Walsh from November last year. In his post (How do you measure technical documents? What metrics do you use?) Ivan describes how a tech writer got fired from the docs team with the justification by management that “technical documents provide no value”.
This is something we’ve probably all come up against at some point in our careers. Fortunately I work for a company where common sense tends to prevail and a well-reasoned argument, from any source, can be influential. But I know of lots of companies where things are very different. Documentation is often seen as just a cost centre. And when the spreadsheet guys are looking at what’s costing money and what’s winning new business it often looks like documentation is something you could do without. Because – think about it – when did good documentation, by itself, ever win one dollar of new business?
The referee was fantastic!
So, how does a documentation manager prove to the numbers guys that they need to spend money on documentation? It’s not easy. And the reason it’s not easy is partly down to the referee scenario. Think of a great football game (and I’m not talking US-type football here, I’m talking about the football the rest of the world plays). With a great game of football you’re never left talking about the referee at the end of the game. In fact after any game of football you never find yourself talking about the referee unless he did a bad job. You rarely notice the referee if he makes good decisions, lets the game flow, nips trouble in the bud, acts impartially … does his job well. You only notice him when he disallows a perfectly good goal, books the wrong person, consistently favours one team, awards a last-minute penalty when it was a blatant dive, and so on.
It’s the same with documentation. People only start talking about the documentation when it’s incorrect, badly written or absent. With documentation: no news is good news. And this is dangerous. It’s easy to cut the documentation effort because initially no one will notice the difference. The quantity and/or quality of output from the docs team will drop off, but because much of the documentation effort is for new product that’s not going to be released for a while – and even once it is released it won’t be the documentation that customers focus on first – and because the docs team will still endeavour to make sure the high priority work still gets done, it’s going to be a while before this change affects customers. In other words you can get away with it for a while before shit starts to hit fan.
The reverse is, of course, true. It takes a while for cuts in documentation resource to work through and start hurting your customers, but it also takes a while to recover from that situation once the powers that be have decided something needs to be done. In fact, after the problem has been recognised things will continue to nose dive, even after you’ve recruited replacement staff, and even supposing the replacement staff don’t need any training and can magically walk in and start being productive from week one. During this time more and more customers will be hit by the documentation-related problems. A head of steam will build up. Customers will talk amongst themselves about how difficult the software is to use. News might even spread to prospective customers. But by this time the damage that was done was done months before – and there’s no quick fix.
It’s back to the referee scenario. Not only do customers never talk about the documentation unless they’ve got something to complain about, but it’s also true to say that while documentation never won a sale, it is sometimes a contributory factor in a lost sale.
This is the point in proceedings – when you’re doing your best to recover from resource-related documentation problems, but customers are still complaining – that necks are on chopping blocks and axes are poised. And you can bet those necks aren’t the necks of the budget police who wouldn’t let you replace a colleague who left, or wouldn’t grow the docs team in line with the growth in the development team, or who “restructured” the docs team to allow budget to be reallocated to more profitable areas of the business.
Docs are important? Go on then … prove it!
This all sounds very grim. So how do you avoid getting into that mess? You might be convinced that what you do is important – but how do you convince others? How do you prove the value of what you’re paid to produce? How do you answer the question Ivan Walsh got asked: “What metrics do you use to tell if the documents are successful?”
Well I’ve been thinking about this and reading around and there’s surprisingly little out there to help me answer this question. There’s been quite a bit written about what metrics you might use to judge individual technical writers – and maybe measure one technical writer against another – or how you can assess the quality of particular documentation deliverables – for example, the number of comments that were raised during pre-publication review, or the number of documentation bugs raised after publication. But the metrics we’re talking about here are measurements of the value of the documentation effort within your company.
I came across a LinkedIn discussion that mainly concerns the micro-metrics of assessing particular documents or writers (rather than the macro-metrics of assessing the documentation effort itself) but these two quotes, I think, touch upon the bigger picture:
in the final analysis any document is only valuable if it answers the questions the user has so they are not required to make some sort of call to a support person
A better measure is the readers' view of the documentation. Do they like it? Can they use it to solve their problems? Does the documentation reduce the number of calls to the tech support center? Does the documentation aid the tech support center in quickly resolving client problems?
The references to support point to a possible solution.
Donald S Le Vie Jr wrote an interesting and useful article in the December 2000 edition of Intercom (“Documentation Metrics: What Do You Really Want to Measure?”), although he mainly discusses writer-specific metrics that he believes you really shouldn’t bother trying to use – for example, hours per page, pages per hour, documents released per month or per writer, errors per page, percentage of time schedule dates were met, etc. However, he ends the article with some metrics he believes are useful:
- Web-based feedback – e.g. find out what customers want more of, or don’t need, in future versions of documentation
- Customer usability of beta or final versions of documentation. Get them to rate the documentation.
- Team up with Customer Support to make sure it’s recorded when support calls are documentation related. “Track this information over time to show the decrease in the number of documentation-related calls so you can assign a dollar figure (a quantitative measure) to any metric.”
Le Vie stresses the importance of regularly broadcasting these metrics throughout management.
Conclusion
In the LinkedIn discussion Padric O'Rouark says:
Sell “help” separate and you would find very few buyers.
That’s true, but it’s also misleading. The fact that you wouldn’t pay for something does not mean it has no value. When I buy a bed from Ikea I expect to find instructions on how to assemble it when I get it home. I wouldn’t pay for those instructions separately though. If I was expected to do this I’d just shop elsewhere. As an aside, the poor quality of Ikea’s assembly instructions is one of the things that makes me dislike the company as a whole (that and the fact they try and steer you round the whole store before you get to the checkout, rather than letting you get in, get what you want and get out).
If customer satisfaction is important to a company then the philosophy of doing only as much as you need to do to make a sale, and not one jot more, probably only works if you’re a make-a-quick-buck-and-disappear business, or if you’re competing solely on price and you’re confident customers will accept poor quality products and bad customer service just as long as the price is right.
Customer-facing documentation has no place in a cut-price or fly-by-night business. But in the software business where companies need time to grow and therefore customer retention is important, user assistance is valuable. If you’re trying to sell corporate software then the people with the purchasing power, the decision makers, aren’t the people who are going to be using the software day in, day out. But if you’re in the business for a few years then some of those people who have to use the software in their daily jobs get promoted and end up being decision makers, or at least decision influencers. If you’ve had to struggle for several years using a badly designed bit of software from Company X, that had little or poorly written user assistance, then, later in life, when you get the chance, you’re going to make damned sure you don’t let Company X inflict any more such pain on any of your staff!
It’s hard to come up with metrics to justify the current size of the docs team, beyond all argument. Bean counters are interested in having more beans to count, so winning new deals and satisfying contracts in order to be paid quickly are things that tend to impress them. In this regard, you could look at the number of documentation requirements within Request For Proposal documents from prospective customers, or you could find out whether documentation was scored as part of customer acceptance tests?
But other than that, what can you come up with to use as proof of your value to the company that pays your wages? Well, the source of your wages may be the key, because really it’s your customers who pay your wages and they are the only ones who can provide metrics that will convince management of the value of what you’re doing.
I started this piece by suggesting that if you sack tech writers, or reduce the proportion of writers to developers, sooner or later customers will start to suffer. If that’s true then it should be measurable. Keep a record of the number of documentation-related support calls or bad customer feedback mentioned in engineer site visit reports. Have numbers gone up or down over the past six months?
You might never be able to prove that customers are happier because of the work you do, but if you’re able to prove that customers are less annoyed with the products and increasingly able to get on with their business without calling support, then you might just be able to convince people that documentation is valuable.
Potentially similar posts
- Deobfuscating the title page – November 2009
- 5 Ways to Make Executives Love the Publications Department – July 2009
- ITauthor podcast #23 – Release notes – December 2008
- ITauthor podcast #14 – August 29th, 2008 – Documentation and Agile Software Development – August 2008
- Switching on the Agile lightbulb – March 2008
January 15th, 2010 at 2:33 am (#)
Hi Alistair,
I've discussed this with some people since then and, for brevity’s sake, here are a few thoughts:
1.Create a baseline – e.g. how many tech support calls/email etc are related to poor documentation? If your company/client doesn’t have this figure, suggest that they explore this.
2.Look at Customer Support forums (see Dell/Sony as an example) and look at where communications are breaking down. Where do customers need more information? Where are they getting confused?
3.Ask the Finance Director, how much it costs to answer 1 tech support calls/email? They’ll tell you. See how you can reduce this number by providing documentation, videos etc that resolves the most common complaints. It’s usually 4 or 5 problems that keep recurring. Escalate these and address.
I think the opportunity for ‘technical writers’ is to look at the business problems within their own organization and then see where/how they can resolve these issues with the communication skills they have.
Scott Able, Chris Brogan, Debbie Weil all offer valuable insight (from different angles) on this.
Bye
Ivan
January 15th, 2010 at 3:18 am (#)
Ah rats, and I was hoping not for the classic, traditional value measures, but for the application of web analytics to tech comm sites specifically. Rachel Potts has a nice article here applying web analytics to the field of technical communication: http://communicationcloud.wordpress.com/2009/11/1...
With web analytics, page views, return visitors, new visitors, how long a reader stays on the page, and the exit rate, these are all metrics that can help you prove (or disprove) your worth as a technical writer. When you have a big-picture view of your help site compared to the support site, community site, corporate site, and marketing site, you can aggregate that info to answer the compelling questions. If you have a lot of pages that are never viewed, well, what's causing that, and how does it affect the company's bottom line?
Hard data such as web analytics also gets you away from the few opinions about the help that overwhelm the discussions with anecdotal rather than hard evidence for one direction for pubs or another.
What do you think? Not enough web content to make the reports stick? Or are we onto something here?
January 15th, 2010 at 9:01 am (#)
Hi Anne,
<If you have a lot of pages that are never viewed, well, what's causing that, and how does it affect the company's bottom line?
We had 5000 pages on a portal many years back. Nightmare to maintain them. Less than 10% were read every month.
This was for several reasons:
1. Users don’t know how to search.
2. The search tool is not up to scratch.
3. The content is not tagged/indexed.
4. Users have only so much patience and then give up (about 3 pages)
We took the 90% offline for 1 month to see what the reaction would be. Zero!
People spent less time on the portal – but here’s the important point – because now they could find the content.
… but they used it more because it delivered.
January 15th, 2010 at 10:16 am (#)
Thanks Ivan - those are all good points. If Support are regularly getting the same questions and Docs aren't doing anything about it that's symptomatic of a serious failure of internal communication.
I work with corporate software where end users call their own Support help desk with their questions and only the big issues, or server-side questions, get asked of our Support people. So I need to reach out to customers and try to get hold of their data from their help desk to find out where the areas of pain are for the end users.
I've campaigned for some time to host our online help on our servers, so that we can collect data showing which parts of the help system users need to look up most often. This would allow us to target those areas and finding out what words users enter into searches would give and indication of which parts of the user interface need attention. Hosted help would certainly make it easier for me to answer the questions: "Where do customers need more information? Where are they getting confused?"
January 15th, 2010 at 10:30 am (#)
That's absolutely right! Web analytics would provide me with lots of great metrics. I guess I missed this just because it's not available to me (not yet anyway). Up until recently all of our end user documentation has been in the form of PDFs and CHM files. We have now moved the online help to WebHelp (output from Madcap Flare), but there is still some resistance to the idea of us hosting this ourselves. Partly this is down to customers in our market sector being wary of information hosted outside of their network - but we do have customers who would be happy to have one less thing to administer.
Again, ultimately it's down to cost. Changes that are cost reductive are always easy to make. Changes that involve spending money - even with the promise of demonstrable value from the results - are much, much harder to implement.
Please keep writing about the value of web analytics to tech comms, and across companies as a whole: it's all grist to my mill!
January 15th, 2010 at 10:32 am (#)
Woh, that's radical!
What did you do at the end of the month - leave the 90% out?
January 15th, 2010 at 12:16 pm (#)
Hi Alistair,
We rolled back about 20-25% over a 3 month period after the content was reviewed & optimized.
Essentially, we convinced the snr mgr that quality v quantity would win… and the feedback backed it up.
But they two key takeways here are:
1. Shoveling content onto a site doesn’t work – it creates traffic jams and information pollution, and you lose the trust of the readers
2. Identify why people use the site and build it from there. if they all read X number of pages, then look at why they are doing so, i.e. is because the user needs more information on the product, policy, procedures etc. or they have an interest in a specific area and are looking for more (unfound) information.
January 15th, 2010 at 12:25 pm (#)
In the book Groundswell, there is a section where they talk about how users’ were asked to ‘rate’ fixes in order of priority.
For example…
If you're Sony and have 1000s of people on your customer support site, you can ask them to rate (prioritize) bugs etc that you want Sony to develop faster than others.
All of this feedback gets funned back to marketing, designers and developers.
You make the fix and release the patch to the customers (or whatever works for your industry)
The customers see that you're listening to them and continue to help you prioritize the development work.
This is a radical change from the approach most companies adopt. You're bringing the customers in close so they can accelerate the dev process.
One downside to this is that it (often) conflicts with internal politics, i.e. people pulling rank over what gets done when but that’s another day’s work…
January 15th, 2010 at 1:04 pm (#)
Your second point again highlights the benefits of hosting web-based user assistance, compared to CHM help files residing on client PCs.
January 15th, 2010 at 1:36 pm (#)
"So, how does a documentation manager prove to the numbers guys...."
Sorry, you lost me at this point. Technical writers shouldn't have to prove or justify anything to anyone. It's part of our job to write useful content, and it's up to us to measure its usefulness. If I was working somewhere where I had to justify every last topic or article to "numbers guys" I'd be very miserable and would be looking for another job. If you don't trust people to do their job, using their experience and expertise, then don't hire them.
January 15th, 2010 at 3:05 pm (#)
"shouldn't have to prove or justify anything to anyone"
:-)
That's some statement! I can tell you you wouldn't get a job at the company I work for with an attitude like that! - but then, from what you say, you wouldn't *want* a job where I work!
;-)
Personally I've never worked anywhere that didn't have an appraisal system in which you were required to talk about the value you'd brought to the company over the past 12 months. Reading through my post again I think I was a bit mean to senior management and the people in Finance. They have a job to do and they're not doing their job properly if they aren't periodically looking at every section of the business and asking questions like: do we need that many people in that department, or: is the company spending its money wisely on those salaries?
If you read my post again you'll see that I never suggest senior management expect us to "justify every last topic or article" - but I do expect, and in fact hope, that senior management should want me to be able to justify the cost of employing me and my team. I'd be concerned if I was working somewhere where I was just left to get on with things and no one ever questioned me or wanted to know about the value of the docs team to the company.
I have a great team of writers working for me and I'm very lucky to have an approachable, common-sense-driven management team above me, but I wouldn't want my efforts on behalf of the company to be just taken on trust. I think there are good reasons why we should continue to work for our company and be paid the salaries we're paid and I'd like to think I can back that up with some hard facts.
January 15th, 2010 at 5:32 pm (#)
The VP Development for the contract I'm working on was looking for a way my service could generate revenue. He said (charmingly), "Tech writers are like toilets: you only miss them when you need one." He suggested I create videos that can be used for both Webhelp and marketing demos. If I'm creating videos for end users, why not make it something the company can use for Sales? The conceptual video idea that @tomjohnson discussed in yesterday's post could cover both angles so I'm going to give it a go.
Is VP Dev right? Do I need to sell myself as a technical and marketing writer to get the work?
January 15th, 2010 at 5:34 pm (#)
If you've seen the movie Val Wilder, you'll know what I mean when I said to myself while reading this post, "write that down!"
Good tips. Thanks.
January 17th, 2010 at 5:46 pm (#)
Maeve,
"Tech writers are like toilets: you only miss them when you need one." That's one I hadn't heard before. I probably shouldn't, but I kind of like it!
:-)
You ask: "Do I need to sell myself as a technical and marketing writer to get the work?"
I hope not.
I regularly edit copy for the marketing department and I've often had to write copy for business proposals (which is a similar sort of thing) and it's a very different skill to technical writing. In my experience technical writers don't often make good marketing copy writers and marketing folks don't often make good technical writers.
February 4th, 2010 at 9:26 pm (#)
A really interesting blog post. I'm a bit late to the conversation, but here are some of the thoughts I've had about this topic:
We all do need to justify ourselves. If you've never had to put together a case for your value to your company, then someone else is doing it for you. Business is (almost always) about generating revenue, so your life is likely to be easier and your job more secure if you can justify yourself in terms of monetary value.
But that's not the whole picture... technical documentation also adds to the overall customer experience, and although there have been plenty of books written about how companies that take customer experience seriously are more successful (Louis Carbone's "Clued In" is a good one), I've never seen anyone put a specific number on the value of a specific customer experience initiative. For example, when luxury hotels choose to use high quality bed linen in their rooms, I doubt they put a specific sum on the ROI for this decision, even though the reason they do it is as part of a set of things designed (ultimately) to generate revenue.***
Instead, the value of these initiatives to a business is often conveyed via stories told about individual cases: an unhappy potential customer who couldn't get the software up and running because required information wasn't available - so they decided not to buy; customers who are so delighted that they tweet about it or write a blog post. Small stories like this can have a powerful impact on how technical documentation is viewed - switching it from being a dirty necessity to something that adds value ... all without needing to come up with any numbers at all.
The reality then is you probably need a combination of well-thought-through metrics AND a few powerful customer stories.
***However, companies *do* often measure overall customer satisfaction, via something like the NetPromoter survey which measures likelihood of a customer to recommend the company's service or product to a colleague, and this is often used as a way to put monetary value on keeping customers happy. If your company is doing this, then it could be a good to measure customer satisfaction with the technical documentation too, and make sure that gets plenty of visibility as something that contributes to overall satisfaction.
February 4th, 2010 at 10:00 pm (#)
Rachel
Thanks for commenting. You make a good point about customers talking about their good or bad experience. Time was companies could afford to piss off some customers (perhaps in order to allow them to suck up to other, more "important" customers). Now that we all have many ways of being heard and broadcasting news of our good or bad experiences, companies really can't afford to behave like that any more. Sam Bayer's blog post "Hertz Gets Social Media!" is a good example of a company being smart about a customer voicing his opinion.
So documentation certainly has a value as a preventative medicine - and like all medicines you accept that it costs money and you have to have a certain amount of trust that things would be worse if you weren't taking it.
But if part of the answer of proving our worth is to arm ourselves with "powerful customer stories", the question is: how do we go about getting those? My experience has always been that, when it comes to documentation, people are very quick to complain when something's wrong, but they rarely go out of their way to shout about it when documentation has really helped them out. I'm not complaining - I behave in exactly the same way.
Is it worthwhile - or acceptable - going out and trying to solicit favourable comment? Or do you just need to make sure that when a customer does say something nice you make sure to ask if you can quote them on it?