I was looking for statistics on BI in the Small and Medium Sized (SMB) market today. It’s surprisingly hard to find material. This blog documents some of my findings and makes a few comments.

First, a Definition of SMB. Techtarget suggests (http://searchcio-midmarket.techtarget.com/sDefinition/0,,sid183_gci1005201,00.html) anyone with less than 1,000 employees is medium sized or smaller. 

Based on my anecdotal experience, one could assume that if you have 1,000 employees, you could make $100M a year in revenue and 100 employees could make $10M in revenue.

If you have $100M in revenue, it’s likely that your annual IT budget is about $3M.  $500k of that might be BI, probably less, but not all of that is money spent on BI software itself, you also have to have your IT staff and the equipment.   I read that somewhere but I can’t find the URL anymore. I think it might have been something O'Reilly wrote (If I find it, I'll add it as a comment).   

If you really wanted to see the benefit of general purpose BI, then perhaps a 10% of your users would need BI.  The rest could probably read the static copies of the report in a web browser or spreadsheet, so I'll ignore those in this discussion.

Can you afford BI?

In 2008, the price for end user BI has been driven down.

Let’s consider a product recently released by SAP/Business Objects.  CrystalReports 2008 costs $495 for the entry level, $895 for the ability to create, and $995 for the ability to integrate with Xcelsius.  If someone is outfitting 100 users, they are going to spend $50,000 just for the software.  I don’t know if that includes all the BI infrastructure, somehow I think not.  I am going to look into that and might report my findings in a separate blog entry (this blog entry is supposed to be about money).  We don’t know what the fees are to install and run it.

I don't think Cognos is so public about their pricing. If I can find a reference, I'll add it as a comment to this (or maybe someone else can).

This means it’s probably a good general purpose number to add to the cost of a Business Objects (BOBJ) or Cognos (COGN) estimate if you wanted to guess at the labour and support costs. It's ignoring the differences in complexity, but at the end of the day, it's probably similar complexity to install CrystalReports 2008 versus Pentaho or Jaspersoft.

Software Licenses + Support and Installation is not the Whole Cost !!! 

The goal for at least SMB is to reduce the cost of each new report or analytic, especially if it is not one you need to run every day or if it is likely to change in the foreseeable future. 

To deliver something in BI,  one has to write a SQL query or create a cube.    But doing this is actually a two step process.  

1.       you must find the data and the relationships between tables, if they exist. 

Maybe the SQL or MDX can do it, but more than likely, it can’t.  If it can’t, then you must design intermediate tables and views and run a series of processes to populate them.   If you’re going for a cube, a cube needs to be built or an existing one modified, and this might even be on top of those intermediate tables and views.  

The costs of doing all this can be significant, but that’s just a start.  The IT deparment will never know when it can drop support for these tables, views, and cubes so it’s probably going to be a permanent investment.

That’s the crux of the problem:  The nature of analytics often means that the need evolves.  It’s either because the analytics identified the problem or the analytic led to other questions.  By their nature, analytics are iterative and sequential. For example:

• I want to see the fastest growing products but it's more complicated than just telling me one measurement.
• for each region, I want to see which products were growing faster than the rolling six month average, at least six times out of the last N periods.
• next, again for each region, of those products that were just discovered, which ones grew by more than twenty five percent from the start of the period to the end of the period.
• I want to know the intersection of those two sets of products across all regions. 
• By the way, I won’t need these questions answered until next year at this time, although I might change them a bit.

That’s an example of one analytic combined with another, iteratively and sequentially. Neither of which would be possible in MDX or SQL without the creation of a complex underlying data structure.  Notice that this isn’t simply an OLAP  “drill down” or a “slice and dice”. These are distinct questions, with the answers needing to be merged and intersected.  This kind of query would take only a few hours in nextanalytics, or a few months in a BI tool.

The problem for the company who wants to use BI is this: Once those products have been identified, the analytic request isn’t needed for another year.   When the request comes through next year, it might be a very different set of questions, so the IT money that’s been spent is wasted. It might even have become part of the infrastructure and can’t be removed without great expense or risk. In the long run, an IT budget is eroded by most analytic requests unless they are the trivial sort you can get from an existing OLAP cube.

The real question is: Why incur such a significant up front and ongoing cost for something that isn’t regularly needed and might need dramatic changes the next time it’s requested?   In my opinion, a business intelligence approach is not really a very good fit for analytics, unless it is very clear that the analytic question is permanent and static.  Even then, it’s entirely possible that the implementation and infrastructure cost is too high to justify many early stage analytic requests where they're not even sure where the questions will lead yet.

Upcoming blogs will consider the roles of dashboards, charting components, spreadsheets, and writing analytic software from scratch.

It Starts By Improving On the Data Access Layer

nextanalytics now has a project on SourceForge.net and at CodePlex.   If you are motivated by true and legitimate Open Source, then go see the project at http://www.codeplex.com/nextanalyticsOS.  The SourceForge.net site is incomplete (circa May 2008) because of technical difficulties in using sourceforge.net, but the codeplex.com site works fine.

The two versions are not the same.  Right now, the commercial version does a lot more.  The reason for the difference is this: According to the root philosophy of Open Source, the community will take the project over, will operate as an entity, and will take the software in the direction that the community wants it to.  In other words, whatever happens is technically independent of nextanalytics corporation.  

That being said, the founder of nextanalytics corporation is a member of that community and will contribute more source code as time and energy permits. 

As consumers, people now have the choice of using the nextanalytics commercial product, or the analytic product from the open source community (according to the terms of GPL2).

ward yaternick is the Founder and CTO of nextanalytics corporation

Are you selling or implementing products from major vendors but still not getting the information you need ?

Did you know that’s been a common complaint for nearly twenty years, back when BI was just taking shape. 

If you've been holding your breath waiting for something to come, I suggest you stop.  Nothing's coming from that direction. They’ve even forgotten that’s what the problem is.  Their big claim (now) is: if you can share something, then that's BI.  In other words, take what you get, share it, and be happy. Oh yes, and their prices went up 10% again this year.

I think I know what you want, but I'd like to hear it from you (if you can spare the time). 

Do you need:

1. low cost and easy entry.  I know you can’t or won’t spend a lot of money to get started. 
2. simple, clear examples. But there are different learning styles, so you need a wide variety.
3. somewhere to go to get advice and answers to my questions, preferably online. 

And, if you need to perform analytic data processing, cheaper and easier than if you did it yourself, then, please, take a look at my product.

And, feel free to tell me about yours, here, in this blog.  I promise to go check it out!

We’ve adopted an un-conventional approach to open source, not that it seems there is any convention to open source.  I'd call it disruptive, but how can you disrupt a disruptive technology?

In the beginning, we designed our software in two tiers.  (more than that really, but for the purpose of open-source, two is a good number ...)

by Ward Yaternick

Where does nextanalytics fit in the BI foodchain?