Sometimes, we make decisions NOT to set a goal, because we don't want to break it.

You might be thinking you really should step up your data quality monitoring- get some data profiling underway to help identify the data domains and areas you most want to tackle in 2012. But you might be also thinking that with all the pressures and cutbacks that many companies are facing, you don't have the resources to implement a full scale profiling and monitoring effort, and so might decide to delay.

Don't wait. Just do it. The perfect is the enemy of the good.

Rather than worrying about how much of your data you are going to be able to cover, or that you can't devote enough resources to tackle all of your reference areas at once, work at the problem from another direction.

First, start with master data.

Master data is the data that all your other data is made from. It's the data everyone uses to view the massive piles of transactional data, so one bad row in a master data table, and the impact is felt across perhaps hundreds of reports, and multiple time periods. If you have a product in the wrong category, then every transaction, across perhaps hundreds of customers, and all time, will be mis-catagorized, and every total, sub-total and calculated metric using it will suffer.

While bad transactions are bad, bad reference data is deadly. Bad reference data takes a good transaction and messes it up.

Worst first!

Make a list of your reference tables/area. Customer, Product, Chart of account, etc. etc. What are the most important for your business? This isn't something I can tell you- you have to think about what is most critical.

If you are a company that purchases large amounts of materials from many vendors, and purchasing decisions are fast paced and critical, then maybe it's your vendor master, and your accounts payable.

On the other hand, if you have lots of interaction with your customers, and errors in the customer master cost you business, then start with that.

The key is to first make the list, and then think to yourself "if I have bad quality data, where am I most afraid it will be?" Start profiling there. You want to find the worst first, and fixing that will have the greatest positive impact.

Get to know your data

Don't worry about setting complex or work intensive goals right away. Data profiling is about data discovery sometimes. You need to wade into your reference data, play with it, tease out patterns and relationships. As you get to know your data, you will be able to better identify where there are issues to tackle, and where root causes might lie for data quality issues.

One approach might be to simply resolve to spend an hour a week, every week, profiling some data. If you aren't do that now, you will find that even just a bit of time set aside will give huge insight- sometimes we get too busy to do the basics, and we miss opportunities to make significant improvements with relatively little effort in our data.

While it's a complex subject, some basic rules of thumb can avoid the deeper rabbit holes.

The first thing to understand about Data Quality rules is they aren't as easy as they may look. Data is in theory something in the ordered world of computers, but in reality is in the "flexible" world of humans. A huge amount of data is entered by members of the group "Homo sapiens" (or mutilated by software written by members of that group) and as a result is not as ordered as we would all like.

The challenge for data quality practitioners is to remove the chaos injected by those highly involved primates (us) and make the data the sterile, ordered, never any question about anything type that we all imagine in our fantasies.

But how?

In the end, it is amazing how powerful and complex the various solutions to this problem are.

But I suggest that there are some basic principles that can help guide us.

First- do no harm.

One of the risks of any data quality initiative is that it actually screws up the data more. Don't define rules that are so complex, and so sure of themselves that they actually make the data worse. Be humble. Don't change data unless you are pretty sure it's a good idea. Err on the side of not screwing up the original. And keep a copy of the original- so if things do go off the rails you can undo- or at least try to understand what when wrong.

Go out and talk to the people

Don't sit in your ivory tower and speculate as to what the data means. Go out there and watch people enter it in. See what real world type things are happening that never make it into bits and bytes.

Attack the basics first

Focus your first efforts on dealing with the basics- they will resolve the vast majority of the issues- don't chase after the outliers until you have the "easy" cases taken care of- the tough stuff is a case of diminishing returns- look first at how to fix processes and train your people to make the majority of typical data entry cases more accurate before you start looking into artificial intelligence based hyper-multi-semantic-algorithmic-learning-matching-holistic-flux-capacitor data quality systems.

Less is more- the fewer rules the better.

So whats the rule about making rules? Try to make less rules, and test them in a pragmatic way. It is possible to have so many rules that the rules themselves have data quality issues- don't go there.

Sometimes the simplest things will bring the greatest benefit.

In the coming weeks, I'll be posting about how to design, implement and monitor Data quality rules using the Datamartist tool.

Obviously, we have to believe something. But today I'm asking you to be skeptical. Question your facts.

Let me give you an example. I'm a Canadian, and looking out my window right now, I can see a pretty healthy snow fall accumulating. Lots of the white stuff. Brings to mind the fact that some cultures in the far north have over 100 words for snow.

Hang on. Is that a fact? Tell me- have you heard a variation of that?

I'm sure I read that somewhere. I've heard others mention it. It makes sense- I mean, people living in the far north would see lots of snow, and would know all about it, and so their language would evolve to encompass lots of different qualities of snow.

Sounds good.

Only, is it? It's an idea that "just makes sense". People seem to just accept it as soon as you say it. People are likely to pass the idea along to others- because it makes a compelling story.

But in fact, it's wrong. I'll let you google to your hearts content if you like to find more evidence than my say so, but after reading a number of articles on the subject, (here is an example, and of course the Wikipedia entry.) it seems clear that there are not 100 words for snow in any language. In fact, English has about the same number of ways of talking about snow as languages from societies in the far north.

So the point of all this is- what myths do you have in your organisation? Things that "everyone" knows are true. Things that when they are explained to you make "perfect sense". Things that you teach to every new hire so that they "know how things are".

The insidous thing about "facts" is that once they gain purchase, any contrary evidence tends to be called an "exception", or discounted.

Use data to find out what is true. Fight to improve the quality of your data to find more and more new truths. Question the status quo if the data contradicts it. Don't assume that something is wrong with the data when "things don't make sense." Maybe they don't make sense because your assumptions are just plain WRONG.

Be very aware that you might be making decisions based on myths that while sounding so plausible, so clear, so common sense, are pure fantasy.

The good news is that all your competitors might be doing the same thing. If you look at your data, and see through it, you might show them all how wrong they are.

Only as the tube of toothpaste starts getting near to its end do I start conserving toothpaste because I know I need to make it last.

Another example is the all you can eat buffet- we eat because it's there and we can. Unlike wasting toothpaste, this has more immediate negative consequences.

When there is lots of something, we tend to use more of it than we should.

When the tube of enterprise storage capacity seems to be always full, and when massive databases make an all-you-can-store buffet the standard mode of operation, very often the tendency is to store everything.

Rather than try to determine what information is of a useful level of quality, or focusing on the key information (and ensuring it IS of useful data quality), we stuff our systems full of every type of field and attribute, with massive bloated forms that are too long for anyone to really fill out properly.

Sadly, this doesn't matter because there are too many fields to check anyways (who can define so many business and data quality rules?), so no one is checking.

If we were forced to make a choice between data A and data B, we might think a bit more about which is more useful for answering key business questions (and by connection, actually think about what the key business questions are).

Instead, how many times have I heard an overworked, rushed subject matter expert say - "Just collect it all, we might need it."

By collecting more, we end up with less.

Now, of course, standards are needed, and for a broad range of tools its counter productive (and horrifically expensive) to let everyone just use what they want. The cost of data centers, integration, etc. will be radically more if a company does not bring order to the chaos and the marginal advantages that a very specific or niche technology might have in one department is often obliterated by the increased support costs and integration issues globally.

But if a company looks at these savings that come from standardization, and extrapolates too far, they can fall off the other side of the benefits curve and find that they're hurting, not helping.

In the article in Harvard business review, the researchers also throw around the term "bitsmith" to describe someone who has both subject knowledge, and the ability to wrangle software, and to even create GASP! custom software that does what the team needs to get done. In many companies, current information technology dogma does not leave much room for people that have the time to be a "bitsmith".

In many companies "Custom software" is a four letter word. Well, I personally have used it, commissioned it and written it and know that often it can provide fantastic value- I've also seen people spend thousands of hours building something that never quite worked when an off the shelf tool a hundred times as good could be bought for a few thousand dollars. It's a matter of being realistic and learning how to see the difference between a problem that is specific to your industry/situation that could really benefit from some custom code, compared to a problem that is huge, main stream, and solved hundreds of times over by existing software vendors.

What I think can happen is a pendulum swing, where a company goes from "no standards, a jungle of wasteful custom software" to "Thou shalt use/buy only software on the following list."

The problem is, making a list that contains everything is just not possible. Things change. Stuff happens. It is possible that a single person might need a single piece of software that allows them to understand something, design something, communicate something that will make that software have a truly massive payback, justifying all sorts of pain and config on the part of technical resources and infrastructure.

The challenge, as always, is to have an open, working relationship between those entrusted with establishing and enforcing standards in terms of tools and those who are expected to use those tools to do business. As with so many things in governance, it's about balance, clear goals, and processes that allow for brilliance, change and creativity while not letting that process become the loop-hole that undermines all those savings standardization brings.

Users taking the initiative with data has been seen as the enemy of a successful business intelligence program.

This needs to change. In a world of ever increasing data volumes and complexity, faster business processes and more data savvy knowledge workers, a purely centralized solution is doomed to fail.

A consensus is starting form that the best architecture is one that blends centralized with more distributed and (gasp) free form, user guided methods. In fact, when we look at what actually exists in most enterprises and take into account the unofficial shadow systems, we're already there, but in two separate camps that aren't talking.

The amount of freedom to allow ranges from letting the users have at it, to opening up the possibility of departmental data marts, but the buzz out of TDWI clearly indicates a growing acknowledgement that a rigid top down architecture is not tenable.

What are Oracle, IBM, Microsoft SAP and SAS (who own more than 70% of the Business intelligence market share) advising as being the right approach?

They advocate big architectures, centralized meta data management, big databases, lots of command and control. They talk about "self serve"- but they mean to existing reports or report interfaces. To be fair, they need to sell the tools they have.

For a refreshing change from this, I very much enjoyed reading Mark Madsens keynote at TDWI "Stop paving the cow path".

We enjoy reading things that we agree with, and I nodded my way through his slide deck.

In his presentation, Madsen points out that centralization won't work, because it:

• Creates bottlenecks
• Causes scale problems
• Enforces a single model

Bottlenecks and Scale

In a centralized system, all requests go into the queue, and the backlog starts piling up.

The size of the department/team that is responsible for making it all work becomes the number one bottleneck.

Are there enough people able to prioritize and analyse the payback on analysis requests? Because in a centralized organisation, the gatekeepers are necessary, and how do they KNOW which requests are the good ones? How does anyone really know?

I'm not sure any company can afford to staff a centralized data warehouse team to be able to handle all the requests as they are generated. Prioritization therefore becomes a single point of failure. Get it wrong, and it can be all wrong. In a more distributed structure, decisions are made at multiple points, some good, some bad, but diversity will often bring more innovative and experimental behavior, resulting in new avenues of analysis that a overly static central team might avoid.

For an indication as to how well users think the central team is listening to them, take a look at how many excel spreadsheets there are around, and how many shadow systems grow like mushrooms throughout the standard enterprise. People think their analysis is important, and even if IT won't or can't they find a way to try to get it done.

In terms of scaling, I can hear the technical types starting to explain about how their servers, infrastructure and approach scales- diagrams and MPP theories pulled out with pride. "Centralizing lets it be scalable- what are you talking about?"

Maybe. But there are traps here too- centralized organisations always want to put everything in one database. Having everything in a single repository starts to become the goal- not the cost efficient analysis of the right data. Not centralizing is very scalable- stand alone machines can just be added for ever.

It may in fact be that data can remain distributed and diverse at certain levels of detail, and more federated approaches can be used, resulting in cheaper hardware and software, and more importantly avoiding a lot of really hard master data management work. Consolidation can sometimes happen at summary levels that make sense from a business point of view- not just blindly following the "one version" mantra.

Enforcing a single model

Isn't having a single data model good? We've been told that it is. In a way, this is the holy grail.

But is there a single, correct, slowly changing model that satisfies everyone in an organisation?

Why do I say slowly changing? Because if there is only one for the entire enterprise, it will change slowly, if at all.

Even if you happen to understand what the right model is, (and by model I mean data model, analysis model, process model, any model) and you manage to implement it while its still the right model, in a year its not going to be the right one. And a centralized, high cost, committed architecture won't and can't adapt. You'll still be paying the mortgage on the data warehouse.

Very large centralized models cannot be comprehensive and up to date, because to be comprehensive they have to be so complex as to be difficult to change, and as a result they quickly become out of date. It's sort of a Heisenberg uncertainty principle for common meta data repositories.

"Giving people their flying cars"

Madsen of course doesn't solve the entire problem in his keynote, but he points out some directions that make sense. And his graphic depicting a happy couple blasting off in their very locally controlled flying car sends the message- users can do their analysis without central oversight or interaction. (Although, one would imagine that some sort of air traffic control would be necessary, and the refueling stations for the cars would probably be run centrally- we're not advocating anarchy here.)

Having built data warehouses, established a data warehouse competency center, and provided business intelligence services for thousands of users, I can testify from first hand experience that centralizing alone is just not going to work. People who worked with me a decade ago will remember the significant amount of time spent creating meta data repositories. Are they still needed? Yes. But they simply can't do everything. Use them with care, and be wary of your ambition for them.

First, accept the fact that users are not mindless consumers. Learn from the fact that they use excel constantly, and they don't just read reports- they build things, adding data, fixing data, re-organizing data. They think. Give them tools that include them as part of the data processing.

Business intelligence cannot not be solely a process where formal requirements are gathered, followed by a publishing exercise of delivering the reports on time.

Are there some reports where this is the case? Sure. Monthly management reports and dashboards shouldn't change every month. The model can work for some amount of the delivered data analysis.

The entire architecture isn't getting ripped out- but if the new architecture is successful in bringing the pent up demand that is currently being satisfied by shadow systems into the light, then distributed, user centric, user driven business intelligence will become a significant percentage of the total.

But the old way of thinking has to change. Don't "Crack down on shadow systems".

Find a way to provide better service, be it self, assisted or centralized service that makes the shadow systems simply a less effective way to do it.

The existence of shadow systems, and the extent of them, is the clearest argument that centralized business intelligence alone is simply not up to the task.

Once you have people doing whatever they want in the self directed part of your architecture, DO watch what they are doing- not to control it, but to learn from it. Everyone constantly re-structuring the customer dimension? Obviously it's time for an update. By watching what users edit, what gaps they fill in, you can find the data quality issues, identify the fuel to put on the self directed fire.

Tools like Lyzasoft, our own Datamartist tool, and Microsoft's Power Pivot in Excel 2010 and others are all going to drive power to the users, and introduce a new balanced approach between centralized and local parts of business intelligence architectures. Visualization tools like Tableau will further give people the ability to create powerful, consumable analysis in a self serve mode.

Will there be challenges with data quality, risk management and wasted time doing pointless analysis? Most likely.

Will the information we gather and the payoff from the successful bottom up analysis efforts make it hugely valuable overall? I for one think so.

We need to learn to trust our colleagues with the data, while at the same time managing the reality of data quality and risk of errors that more free form techniques can create.

Companies that include both top down and bottom up capabilities in their architecture will stop wasting time fighting internally, and start to take advantage of all that data.

• $150 000 is the AVERAGE cost of business intelligence software for a DEPARTMENT
• ETL software (Extract transform and load) is also $150 000 on average.

And the rule of thumb for cost of effort and services is 5 times the software cost

I'm not making this up. Check the link.

In the end, Boris suggests that the cost of a single, fairly straight forward report might be $20,000. Of course as he rightly points out there are lots of variables, and it's a classic case of "it depends", but even so- clearly you want to be sure the reports add value when you are using a process that requires that kind of investment.

Boris mentions in passing that the cost of a single day of an external developer he uses for estimating is $800 USD. You can buy two licenses of Datamartist and take a friend out for dinner for that.

Don't get me wrong- for a number of applications you need the big enterprise stuff- but in my mind it makes sense to avoid it when you can. Enterprise business intelligence has its place, but there are alternatives. The rampant use of Excel spreadsheets is evidence of the fact there is huge demand for data out there. Try Datamartist and find another even more powerful way to get a the data for those cases where you need to do more than a spreadsheet, but it's not time to kick off a data warehouse project.

In the data analysis realm, spreadsheet errors are often used to try to dissuade companies from letting their people "work with the data directly". Software vendors of all sizes, but particularly the really big ones (those incumbants) spread FUD because if they can stop people from getting at the data themselves, it increases the chance of companies buying some more business intelligence suites.

The argument goes something like this:

Spreadsheets have been shown to be plagued with errors, many studies showing error rates above 90%. You need to reduce the risk that spreadsheets are creating in your organization by establishing formal, documented processes that are created an managed by professionals using sophisticated tools.

Then the usual nightmare scenarios are brought out, all involving rabid Auditors, Sarbane-Oxley, governance failures etc.

Now, don't get me wrong, spreadsheet errors are a very real and serious problem, and there are all sorts of data applications that should never be done in Excel or other ad-hoc, user driven tools. Ever. Formal documented processes are critically important, and there are lots of places where you better be using the right tools and professionals.

I have seen the culture of the spreadsheet completely undermine initiatives that would have driven better data quality, data analysis and business processes. The spreadsheet certainly has its dark side.

But the problem is that FUD paints with a broad brush. People take it as "Spreadsheets with data in them? Bad news. Don't do it. Individuals able to get at the data, and quickly transform it, analyze it? Who knows what they'll do- shut them down!"

Sadly, from a data quality point of view, sometimes the spreadsheets have the BEST data quality- because people have fixed the issues they can't fix in the transactional system due to constraints or IT department delays.

Encourage positive change with reasonable controls.

Intelligent, responsible people should be encouraged to use "informal" methods and tools to do data analysis.

These people will find things, learn things, and drive positive change (including change in those big formal professional systems).

They should do it with a reasonable understanding that doing things in an informal way, with spreadsheets or other tools does introduce errors, and should consider this when they recommend taking action based on the results.

Balance between two extremes

The totalitarian state: I don't think there is an IT department in the world that is capable of stopping all unofficial data analysis. In fact, I would suggest that the moment such an IT department comes into existence, it would kill the host company, a harsh sort of self-regulation. People interested in data and thinking for themselves would just pack up and leave. So who would be left making the decisions and based on what?

The twisted web of spreadsheets: Companies that allow an anything goes, visual basic code, macros and manual cut and paste direct to the annual report environment are not going to be long for the world either. They populate the horror story pages on the spreadsheet risk websites.

The zone of win.

You want to be somewhere between insane spreadsheet addiction and strict formal big tool paralysis.

I submit that companies that balance risk while still encouraging their smart people to "play" with the data and do analysis in new and interesting ways with new tools are going to win.

Again, don't let this process generate your profit and loss statement- understand where and what the informal discovery process is for- but do let it discover things. If it discovers something interesting you'll have the chance to check for the errors. Make sure its part of the process to do so.

By letting the FUD get you down, you'll never get that far and who knows what insights you might be giving up?

Of course, we believe you should go even further and give those intelligent, responsible people new tools that are less error prone than spreadsheets but still provide as much or even greater flexibility. That's why we're building Datamartist after all.

Openness, balance, and clear minded pragmatism will get you further than FUD every time.

IBM made a major move into data analytics by establishing a new organisation "Business Analytics & Optimization Services" with 4000 people in it.

There was the much quoted Hal Varian of Google who predicted that the sexy new job this century will be some sort of data analyst/statistician.

But I believe there is a powerful force in many businesses that will slow down our headlong rush towards a fact based, analytical thinking, data quality focused future.

As a group they are generally referred to as "Upper management" or "Leadership".

Now to be fair, there are obviously great leaders and executives that understand that data is important.

But the fact that making decisions based on facts and data is actually defined as school of thought- "Fact based management" or "Evidence based management", or in the medical area its called "evidence based medicine" illustrates that too many alternatives still exist.

The lies and dirty tricks of anti-data leadership

They make comments that equate analysis with "delay".
They confuse considering options with "indecisiveness".
They don't invite people who actually have seen or understand the data to their meetings.

They come up with all sorts of alternate ways to make decisions- and defend their position even when the data clearly does not support them:

Call it strategic

I know the numbers don't add up right now, but this is strategic.

What does that mean- our strategy is to do things without ROI?

Go with consensus perception

We don't have time to get the actual data- we're going to have to make a decision based on what the people on the ground are seeing.

If you ignore data, create a hypothesis and then go looking for supporting "evidence" in the form of people "on the ground" thinking it's a good idea, you'll find it.

People take suggestions from your questions and generate a matching memory/perception of what they think is happening in the real world. This is something that is well understood and the accuracy of eye witness testimony is known to be poor.

Blame the data quality

You know we have issues with that data. I don't think we can risk relying on it.

So what's the alternative? Tea leaves? Might be some risk in that too.

And why is the data quality an issue? Probably because leadership didn't approve the budget and support the process changes that would have improved it. If the top executives aren't responsible for data quality in their organisation and have decided not to use the data then a company is in a sad, dysfunctional state.

Moving forward- fight the anti-data forces of evil

Now, no-one can analyse forever- eventually a decision needs to be made.
Often, not all the analysis we want to do can be done. The number one reason anti-data leadership will likely reject doing detailed analysis is that it takes too long. They want to "pull the trigger" and get going, even if the decision is clueless (literally).

Always be working on fixing the structural issues that slow analysis down

These kinds of issues can slow you down:

• If you have bad quality data in your systems, any analysis must first fix it- causing delays.
• If you don't have the people on staff to do the analysis, you have to hire consultants, adding delay and cost.
• If your data definitions are inconsistent across the company and with industry standards, mixing data from between operating units and other data sources takes forever.

Create a culture of data

Some examples of beliefs that need to be openly stated and shared:

• the best way to make decisions, if possible, is by looking at actual data.
• firing off decisions made on the basis of hunches isn't being "aggressive and decisive". It's sloppy and incompetent.
• data management and analysis is a key competency for ALL employees in ALL departments not just information technology.

Create data analysis SWAT teams

On top of this, there are new techniques needed to enable data analysis to be fast enough to make decisions timely. It is just not possible to launch a waterfall project, to try to find a date three weeks from now when everyone can get together for a functional requirements meeting.

Companies need to create teams (perhaps virtual, coming together when needed) that are able to use fast, flexible tools to do analysis quickly. I am hoping that the Datamartist tool is one of the new tools that such SWAT teams would have in their toolkit.

The bottom line is that companies who have leaders that "get" data are going to be running circles around companies with executive dinosaurs who's eyes glaze over if anyone starts actually talking about facts and figures that can't fit on a single three dimensional pie chart in power point.

The future is data, but can we overcome the anti-data forces and their dataphobia?

]]> http://www.datamartist.com/anti-data-leadership-the-lies-of-non-fact-based-management/feed 3 http://www.datamartist.com/business-intelligence-adoption-low-and-falling http://www.datamartist.com/business-intelligence-adoption-low-and-falling#comments Wed, 20 May 2009 15:57:05 +0000 James Standen http://www.datamartist.com/?p=2257 It seems that the level of adoption of business intelligence tools as a percentage of users is much lower than typically thought. The data warehouse institute (TDWI) published a commentary on the latest business intelligence survey.

This survey, published by the business application research center, reports that although your BI vendor might be telling you that baseline adoption is 20%, it turns out it's a lot less.

In any given BI-using organization, notes Nigel Pendse, a principal with BARC and the primary architect of BI Survey, just over 8 percent of employees are actually using BI tools. Even in industries that have aggressively adopted BI tools (e.g., wholesale, banking, and retail), usage barely exceeds 11 percent.

I have to admit that the around 10 percent didn't surprise me overall, but I was expecting that in those particular industries they would be much higher- i.e. even higher than the 20% figure that big BI has been telling us for years. Not having worked in retail or banking directly, I just assumed that they were all glued to their reports and cubes.

The article goes on to say that some indicators show that adoption is actually falling slightly. All this while the mega vendors have been focusing their message (and supposedly the capabilities of their tools) on "BI for the masses" and "self serve business intelligence".

So what does this mean? Well, maybe it means that all those people who are frustrated by their big business intelligence, are disappointed at how few people are actually using the reports, and are wondering how the other companies do it should know that, well, on the whole the other companies don't do it either.

On the other hand, maybe the "right" number of people using business intelligence is 5%. Maybe in some industries it's three people at head office and thats it. In other cases it might be that the target should be every single employee using BI every single day. I think in the end the ever useful "it depends" is in play here.

What I do believe, however, is that big business intelligence is broken (I'm talking about the mega vendors). If it's not completely broken, then it looks broken and everyone is talking about how it's broken. Which is just another form of being broken.

I wonder what would happen if we looked at the adoption rate for Microsoft Excel for data analysis? Going out on a limb here, but I'm betting it's more than 8%.

People vote with their behavior. Obviously, its not all doom and gloom, and it is possible to have a highly successful business intelligence deployment. In a former life, I had issues making sure we had enough funding for Cognos licenses because certain cubes were so popular that we had more demand than supply. But I've also looked at log files for entire data marts that were flat out empty. Hundreds of reports built and maintained without users.

What makes BI successful

Generally, the key difference wasn't the tool, or the features, it was the content. Data warehouses and data marts that had been driven by the business and were focused on what the business wanted to see, in the way they wanted to see it were successful. Projects that were technology focused, run by IT and did not have the needs of the end users anywhere on the map would end up doing nothing but consuming cash and data center space.

It simply does not matter if the report takes 2 seconds or 2 hours to generate if it is not important to the business and there is no decision to make after seeing it.

Big business intelligence seems to think that BI for the masses is a tool problem- something in how their portal works, or how many rows of data per second their appliance can process. Sure, if the tools are hard to use or learn, it's a factor, but I think more often than not business intelligence isn't used because it's not providing what is required.

There are a lot of very talented business intelligence professionals that work very hard to deliver the goods, and there are lots of very successful business intelligence projects of all sizes that create real value. But the whole industry needs to take a hard, honest look at where we've come from, where we are going, and be honest about who is using what and why.

Often, people use excel because last week they didn't know exactly what they needed, and it is a tool that lets them build it themselves this week when the boss wants the answer and there is a decision to make. With all its flaws, it's still the most adopted Business Intelligence tool in the world.

Should everything be done at lightning speed? Is it really never possible to know what analysis is needed and use a traditional business intelligence approach to creating it? Of course not.

But are there some types of analysis that need to be done very quickly? Yes. Should we as an industry ignore these, or "leave that to the users in excel". I don't think so.

I'm spending all my time building a self service data transformation tool because I believe there is room in the overall architecture for less formal and more creative, ad-hoc tools. Excel is the ultimate in informal (often far too informal). At the other end of the spectrum, a carefully run, tightly managed data warehouse project is the ultimate in formal (and often is overkill, hopelessly expensive and too slow for a rapidly changing environment).

I'm looking at the middle ground where we meet the users part way, and nurture capabilities to do rapid prototypes, one-time analysis and user driven data transformation. Not as a replacement for big BI- but as another tool in the tool box.

]]> http://www.datamartist.com/business-intelligence-adoption-low-and-falling/feed 5