Making Sense of the Technology Matrix, Part 2: Picking a Winner

In Part 1 of “Making Sense of the Technology Matrix,” we discussed how to narrow the field of apps available to you and your clients to those that will have the biggest impact. It starts with choosing which accountant workflow processes bother you, identifying your infrastructure priorities, narrowing down the list with what integrates with QuickBooks® Online and ultimately doing time-boxed reviews of the remaining set of apps. Part 2 will focus on the final five steps to ultimately pick a winner.

Step 6: Determine your top 5 criteria of evaluation. From the time-boxed evaluation, you now have a sense of which apps to further consider and which ones to eliminate. More importantly, you also have formed an opinion on what criteria you would like to use to evaluate the available technology. So, now it’s time to document.

First, make a comprehensive list of the evaluation criteria. Common choices include:

  • ease of use
  • total cost of ownership
  • customer service
  • workflow fit
  • reporting
  • integrations
  • scalability
  • performance
  • switching cost
  • product training 

Then, narrow down the list to your top 3 to 5 criteria only.

Step 7: Do a 100-point exercise to weight each criteria. Now, you need to weight each criteria to your order of preference. With 100 points to spend, allocate points to each criteria based on importance, such that the points add up to 100, with no score more than 100 and no score less than 0. Ideally, you don’t want to have evenly weighted criteria. An example of four weighted criteria might be 40, 30, 20 and 10 points, respectively. By doing this, you can develop a mathematical equation to your qualitative analysis, removing any biases you might already have.

Step 8: Score vendors of interest across your evaluation criteria. With your criteria set and your vendor list previously trimmed, we want to score each of them on each evaluation criteria individually, using a Harvey-Ball style analysis. Essentially, each vendor will be given a score between 1 (poor fit) and 5 (excellent fit) for each criteria. For example, if cost was a criteria, the cheapest would be given a 5 and the most expensive a 1; for ease of use, the most intuitive a 5 and the most difficult a 1. You should score each criteria across all vendors vs. each vendor across all criteria. This helps you to adjust your scoring by comparing the vendors against one another (you should force yourself to have different scores for each vendor in the same criteria.).

Step 9: Calculate the final score to determine your winner(s). With your matrix of vendor scores, you have created a simple calculation for a weighted comparable score across vendors. If, for example, you had identified three criteria (A, B, C), weighted them (Wa, Wb, Wc) and scored them across two vendors (Sa1, Sb1, Sc1 vs. Sa2. Sb2, Sc2 respectively), your comparable scores would be:

  • Score (Vendor #1) = Wa * Sa1 + Wb * Sb1 + Wc * Sc1
  • Score (Vendor #2) = Wa * Sa2 + Wb * Sb2 + Wc * Sc2

Scores will range from 0 to 500 (or 0 to 5, if weighting is set as a percentage). Sort the list of vendors from largest to smallest. The vendor with the largest score is your winner.

If you want a template, check out this Karbon Technology Prioritization Google Sheet that I created, which is already set up to use.

Step 10: Sign up and conduct a time-boxed free trial, using your firm’s data. With your winner identified, you need to test whether it is right for you and your firm. Register for a free trial, upload your practice's data and try it out for a pre-determined amount of time (one or two weeks). Did it solve your problem? Did it live up to your expectations? Was your preliminary evaluation correct? If yes, then congratulations! If not, look at the second place winner and repeat. In some cases, if two vendors are close in scores, both should be fully tested.

In summary, the ten steps to adopting new technology are:

  1. Start with what bothers you, your team and/or your client.
  2. Determine your infrastructure priorities.
  3. Review the collective set of available technologies for each given priority.
  4. Check apps.com to narrow the list to only those that integrate to QuickBooks.
  5. Do a cursory time-boxed evaluation of all the vendors of interest.
  6. Determine your top 5 criteria of evaluation.
  7. Do a 100-point exercise to weight each criteria.
  8. Score vendors of interest across your evaluation criteria.
  9. Calculate the final score to determine your winner(s).
  10. Sign-up and conduct a time-boxed, free trial, using your firm’s data.