Research and analysis organization Evaluator Group releases a variety of data management and storage supplier reports. We took a closer look to see how it differs from other consultancies such as Forrester, Gartner, GigaOM and IDC.
One area it looks at is managed backup-as-a-service and its clients can compare different vendors, up to four at a time, on a feature-by-feature basis. This is called an EvaluScale report. We saw an example of such a comparison, looking at Carbonite, Commvault Metallic, and Datto. This listed around 60 features grouped in categories such as backup sources, backup targets, efficiency and performance features, recovery, and high-availability.
We asked managing director Camberley Bates about the EvaluScale report, the group and its focus.
Blocks & Files: The Managed Backup-as-a-Service comparison report is more informative than competing reports based on supplier documentation alone. I guess the Evaluator Group goes further than that?
Camberley Bates: We do our own research. We do not have or request the vendors fill out a form [or] questions. We find the data sourced in this manner can be misleading and very much marketing-speak. Thus our research comes from:
- Reading technical manuals, admin guides
- Vendor technical and business briefings
- Technical review and clarification with the vendor (usually product managers)
We allow non-subscribers (and of course subscribers) to review and comment before we publish.
The criteria for EvaluScale comes from research with our IT end users and our engagements with them.
Blocks & Files: This is good in-depth competitive information at a feature level. Is it used as such by suppliers, as well as by customers looking for a feature-by-feature product/service comparison?
Camberley Bates: Correct, our client base includes vendors, IT end users (large firms that typically have a Gartner license, but are looking for more depth and advisory services), and channel partners.
Blocks & Files: There is obviously a database behind it. Did the Evaluator Group build its own? How often is the data in it per category (managed backup services etc.) refreshed?
Camberley Bates: Yes, this is built on our own [database]. We have been producing these matrices, which at first were Excel sheets/PDFs since 1998. They are now built on a dynamic system. The data is refreshed with product announcements, most times within the week of the announcement. However, often new products may drag out as we do not have complete data or the vendor did not get us briefed prior to the announcement. We also go through all the matrices at the end of the year and update, to make sure we have caught all the announcements/changes.
There are a few matrices that are dragging behind right now – for instance, the managed BUaaS and the financial-as-a-service (consumption models) due to different priority resources.
Blocks & Files: Does the Evaluator Group provide supplier product/service ratings based on counting and scoring the features? Like a GigOm Radar feature matrix, for example.
Camberley Bates: Yes and no. We score features as they pertain to the criteria. There is no “counting.”
Blocks & Files: Does the Evaluator Group provide market supplier summary diagrams like Gartner’s Magic Quadrant and similar diagrams from Forrester, IDC, and GigaOm? What’s the group’s thinking on such diagrams which are based, in part, on supplier product/service feature scores?
Camberley Bates: We released one absolute ranking and that was for the Container Management Platforms/Services. I don’t know if we will do more. Internally we run into the classic engineering problem – “the rank depends on the requirements of the organization.”
For instance, when doing the analysis on CMP, the decisions were highly dependent on the business drivers and priorities. In the process of the research we identified five different high-level business drivers that changed the decision outcome. For instance, “infrastructure Leverage” – an integrated management approach for IT Ops.
Blocks & Files: How does the group differentiate itself from Gartner, GigaOM, Forrester, and IDC?
Camberley Bates: We are highly technical and have a very pragmatic view of the IT end user – the guy on the front lines.
Blocks & Files: Does this show up in product testing results?
Camberley Bates: We knew from personal experience a particular product that, one, could not scale and significantly dropped off in performance, and, two, was extremely difficult to install. We had tested it in the labs. To get it running we had to go back to their support. We finally got a person to tell us: “Well, it does not work that way.” And despite the tuning, with their assistance, it still fell off in performance. Unfortunately, we cannot test everything and this is not a corner case. It happens in about one in five testing projects we work on.
Blocks & Files: How much do paid subscription plans cost?
Camberley Bates: IT end users – $7,500 to $21,000 a year. Vendors start at $25,000 for small firms and scale into low six digits.
