CharityVillage® Research: The problem with best practices in technology

Resources & Library

Path: Main Street > Resources/Library > Research Articles > Feature Article

The problem with best practices in technology
By Gillian Kerr, RealWorld Systems
The information in this article is current as of January 11, 2002.

The term best practices is often misused in talking about the uses of technology. Best practices can be a powerful research method that identifies successful approaches in any field. In practice, best practice research often ends up being a collection of cute stories about organizations sharing their interesting ideas. The ideas are not examined critically or evaluated for effectiveness.

This is a serious problem for agencies that are trying to decide about investments in technology. Many technology programs may seem like a great idea at the time, but lead to disaster or prove to be a giant waste of time after a couple of years. Before accepting any best practices at face value, it's a good idea to determine what evidence underlies them. I suggest dividing them into three categories:

1. Cute stories. These are ideas, programs or pilot projects that have interesting features for other organizations. They can be used to help agencies to brainstorm, or to think about options that may not have occurred to them otherwise. They can also suggest programs that are worth studying in more depth. In that sense, they provide a useful function. However, almost none of these ideas have been critically evaluated, and many of the programs are abandoned soon after they are introduced. For some terrific examples of this approach, see the Benton Foundation's Best Practices Toolkit. Some of the Benton articles include deeply valuable insights drawn from experience, and are worth studying in more detail, as well as replicating across the sector. Some don't. The Benton Foundation's collection is one of the best available, because they carefully screen the stories included in the site.

2. Generally accepted procedures. Like Generally Accepted Accounting Principles (GAAP), this type of best practice lays out professionally expected standards of behaviour. Npower is planning to provide a list of best practices in nonprofit uses of technology in their report, due to be posted February 2002. It will include recommendations for data security and backup, networking, email policy and so on. The report aims to represent a consensus of acceptable technology procedures among nonprofits in the U.S. When this type of best practice builds broad consensus among experts in the field, it becomes a standard that, if not followed, opens organizations to legal action or public relations risks when things go wrong.

3. Successful models. This type of best practice is the rarest because it's so difficult to identify. One of the best examples of best practices research that I've seen in years is Jim Collins' new book, Good to Great: Why some companies make the leap and others don't. (See this article in Fast Company for a summary of his research.) Collins identified eleven U.S. companies that had made a dramatic and sustained improvement from mediocre to outstanding financial performance, with the excellent performance lasting at least 15 years. The criteria for excellence were so strict that only 11 out of 1,435 publicly traded companies met all of the criteria. Then Collins and his team investigated those 11 companies in detail to try to understand the factors that made them successful. They found some fascinating behaviours that separated these successes from their mediocre competitors. The book is worth reading in full, partly because Collins believes that these elements characterize successful nonprofit organizations as well as private companies. But my major point is that good best practice research requires a careful process of testing to ensure that you are coming up with true best practices as opposed to a temporary burst of good results.

According to Collins, great companies invest in technology differently from other companies. In fact, this was one of the seven key points that characterized the most successful organizations. "They never use technology as the primary means of igniting a transformation. Yet, paradoxically, they are pioneers in the application of carefully selected technologies." (pg 13). Organizations must ask themselves two questions: Does the technology directly support the factors that are key to its mission and strategy? "If yes, then you need to become a pioneer in the application of that technology. If no, then ask, do you need this technology at all? If yes, then all you need is parity. (You don't necessarily need the world's most advanced phone system to be a great company.) If no, then the technology is irrelevant, and you can ignore it." (pg 153).

The nonprofit sector desperately needs good research to identify what technologies support the key activities that make them effective. Many promising tools, such as online fundraising or volunteer matching, have not yet proven themselves, but will probably have a profound effect on the sector once the bugs are worked out. Until then, they are cute stories. Other technology tools, such as email and online research, have offered dramatic improvements in communication and service delivery, especially among rural agencies. Those tools, like telephone systems, will soon be "generally accepted standards" for professional organizations.

Best practice research among nonprofit organizations will be difficult to do. Collins chose publicly traded companies in the United States for his research because he could base definitions of high performance on decades of publicly available detailed financial statements for more than 1,400 companies. Financial success in for-profit companies is clearly a vital element of overall success, even if you take other components (such as good citizenship) into account. But for nonprofits, how do we define a group of hugely successful organizations so that we can study them? How can we measure long term performance among a wide range of different nonprofits, including arts, sports, human services and education? Universities are often compared using a variety of measures, including peer review process, and while there are always problems with comparisons between very different organizations, it provides an interesting basis for best practice research.

The other major approach to best practices is to study specific programs and investments to see which ones are successful over time. For example, one could track the effectiveness of online fundraising programs using appropriate measures like return on investment, market penetration, customer satisfaction, and so on. This approach is essential for understanding and improving particular tools, and is just as important to the sector as the broader research on what factors lead to effectiveness as whole organizations.

In summary, the evidence for effectiveness of technology investments will be difficult to gather and assess over the next few years. Best practice research is one way of approaching the question - but be careful in evaluating the results.
************

Gillian Kerr, Ph.D., C.Psych.

President, RealWorld Systems

gkerr at realworldsystems.net
Read my weblog at http://blog.realworldsystems.net