Gartner and Burton Group SaaS Surveys: Same High-Level Findings, Different Conclusions

Blogger: Craig Roth

Gartner just published a survey that “Shows Many Users are Underwhelmed by Their Experiences of SaaS.” I ran the survey Burton Group did on SaaS with Ziff Davis Enterprise, so I was interested to see how the stats and conclusions compared. Our high level stats are pretty comparable, but our survey asked those one or two more questions that changed my view of the findings and resulted in some very different conclusions.

Quick methodology note: Gartner’s SaaS survey was of 333 enterprises in the U.S. and U.K. in December, 2008. Burton Group’s SaaS survey was of 318 enterprises in the U.S. and Canada in April, 2008 and was supplemented by an executive panel discussion. This means our data is 8 months older than Gartner's, which could make a difference since the financial crisis hit right inbetween.  I haven't seen the full survey doc from Gartner since it's client only (our full report is currently available for free download) so I can't compare the demographics of large/small companies and roles to see if they could account for different responses.  But the fact that our results were very similar in some areas suggests they may be close.

Some interesting comparisons:

• Their survey found organizations are “somewhat satisfied with SaaS … (4.74 on a 7-point scale).” That’s almost exactly what we found (4.60 if you translate our 4 point scale to 7 points). But satisfaction compared to what? Meeting business needs with software is tricky whether you do it with a SaaS or conventional distribution model. You have to ask one more question, which is how satisfaction compared to the main alternative (conventional in-house software).  We did ask that question and found 21% thought it was superior compared to 12% inferior (the rest found it roughly equal). So when you ask “is it better?” rather than just “how did you like it?” SaaS comes out looking better in comparison to the status quo.
• Gartner asked for “the top three factors that they would consider in making their decision to deploy SaaS” and found “meeting technical requirements was the top overall consideration at 46 percent, followed by security, privacy and/or confidentiality at 33% and ease of integration and functionality needed for business unit owners, both at 29%.” Our respondents said the top 3 pros were no in-house maintenance (57%), shorter rollout (49%), and usable anywhere via internet (46%).
• We both asked companies that had decided against SaaS "why not?". Gartner said “42% cited high cost of service, 38% said difficulty with integration and 33% percent said the solution didn’t meet technical requirements.” Our study showed information security for 48% of respondents, integration for 40%, and didn’t meet ROI requirements for 38%. I'm surprised information security didn't show up on Gartner's top 3 as it was the #1 issue for our respondents. In fact, I would think information security became only more important since we gathered our data.
• When Gartner finds “SaaS is not quite the panacea it often promised to be”, I have to ask who is doing the promising? Sure, some vendors go overboard on promising the world, but a lot of the promising is being done by journalists, SaaS evangelists (often within enterprises), and (dare I say it) analysts.
• Gartner stated “These findings contradict the general impression that SaaS could help alleviate costs …” Again, who is spreading that impression? Our survey found implementers were more interested in trying to skirt IT’s priority list than save money. Then Gartner reports goes on to state “Ms. Lo said that SaaS vendors must focus on truly delivering lower TCO …” Certainly, but how does that compare to expectations? We addressed the TCO question with our respondents. First we asked how TCO of SaaS compares with conventional software. 33% thought it was worse (somewhat higher), 24% the same, 26% better (somewhat lower). Doesn’t sound great, but ask one more question: did SaaS deliver the TCO you expected? An astounding 95.5% of respondents said “yes”! That’s because they were smarter than pundits give them credit for. They knew SaaS wasn’t like an ATM machine and it met their expectations – remember, they weren’t in it for the money in the first place.

All this reiterates the key findings we published in July of last year and unveiled at our Catalyst conference:

1. SaaS implementations aren’t all about saving money. They are mostly about avoiding IT, quicker rollouts, and adding predictability/clarity to pricing
2. SaaS implementers haven't saved a ton of money, but they didn’t expect to either – see #1
3. Sure SaaS satisfaction is just lukewarm, but that’s because complex software for complex (or undefined) business processes is a really tough thing to get right no matter how you do it. You have to compare to conventional in-house software to see if it’s the lesser of evils
4. Go beyond the stats – we did a panel in conjunction with the survey which pointed out nuances and feelings about SaaS that clarified the stats. Those stories clued us in to how frustrated end users are in dealing with the priority lists, long rollout times, and unpredictable costs of IT. If IT took on tasks instantly (without requests, budget cycles, and prioritization against other projects you don't care about), got new systems rolled out in a few weeks, and gave an exact cost estimate up front, you'd see a lot less interest in SaaS.  So for those avoiding the media spin, avoiding IT (even for buyers already in IT that don't want to deal with one more complex project) is what gets people excited about SaaS, not saving a few bucks.

Comments

[Jeff Shuey]

Good read. Thanks for posting this and summarizing the key points. I think your assessment of the two different points in time for The Burton Group and Gartner gathering research data are indicative of the market that surrounds SaaS. Meaning, SaaS has to ride the same business and economic waves that everything else does. Over time I do thing the concept of SaaS will shift and continue to gain tractioin and acceptance in the enterprise. It will probably get a new name too.

[coach purses]

that's not where you want to be
everytime you hear the rolling thunder
You turn around before the lightening strikes
And you could find a rock to crawl right under

[Term Papers]

Reading this made me feel smarter,better informed.Great post!