Users taking the initiative with data has been seen as the enemy of a successful business intelligence program.

This needs to change. In a world of ever increasing data volumes and complexity, faster business processes and more data savvy knowledge workers, a purely centralized solution is doomed to fail.

A consensus is starting form that the best architecture is one that blends centralized with more distributed and (gasp) free form, user guided methods. In fact, when we look at what actually exists in most enterprises and take into account the unofficial shadow systems, we're already there, but in two separate camps that aren't talking.

The amount of freedom to allow ranges from letting the users have at it, to opening up the possibility of departmental data marts, but the buzz out of TDWI clearly indicates a growing acknowledgement that a rigid top down architecture is not tenable.

What are Oracle, IBM, Microsoft SAP and SAS (who own more than 70% of the Business intelligence market share) advising as being the right approach?

They advocate big architectures, centralized meta data management, big databases, lots of command and control. They talk about "self serve"- but they mean to existing reports or report interfaces. To be fair, they need to sell the tools they have.

For a refreshing change from this, I very much enjoyed reading Mark Madsens keynote at TDWI "Stop paving the cow path".

We enjoy reading things that we agree with, and I nodded my way through his slide deck.

In his presentation, Madsen points out that centralization won't work, because it:

• Creates bottlenecks
• Causes scale problems
• Enforces a single model

Bottlenecks and Scale

In a centralized system, all requests go into the queue, and the backlog starts piling up.

The size of the department/team that is responsible for making it all work becomes the number one bottleneck.

Are there enough people able to prioritize and analyse the payback on analysis requests? Because in a centralized organisation, the gatekeepers are necessary, and how do they KNOW which requests are the good ones? How does anyone really know?

I'm not sure any company can afford to staff a centralized data warehouse team to be able to handle all the requests as they are generated. Prioritization therefore becomes a single point of failure. Get it wrong, and it can be all wrong. In a more distributed structure, decisions are made at multiple points, some good, some bad, but diversity will often bring more innovative and experimental behavior, resulting in new avenues of analysis that a overly static central team might avoid.

For an indication as to how well users think the central team is listening to them, take a look at how many excel spreadsheets there are around, and how many shadow systems grow like mushrooms throughout the standard enterprise. People think their analysis is important, and even if IT won't or can't they find a way to try to get it done.

In terms of scaling, I can hear the technical types starting to explain about how their servers, infrastructure and approach scales- diagrams and MPP theories pulled out with pride. "Centralizing lets it be scalable- what are you talking about?"

Maybe. But there are traps here too- centralized organisations always want to put everything in one database. Having everything in a single repository starts to become the goal- not the cost efficient analysis of the right data. Not centralizing is very scalable- stand alone machines can just be added for ever.

It may in fact be that data can remain distributed and diverse at certain levels of detail, and more federated approaches can be used, resulting in cheaper hardware and software, and more importantly avoiding a lot of really hard master data management work. Consolidation can sometimes happen at summary levels that make sense from a business point of view- not just blindly following the "one version" mantra.

Enforcing a single model

Isn't having a single data model good? We've been told that it is. In a way, this is the holy grail.

But is there a single, correct, slowly changing model that satisfies everyone in an organisation?

Why do I say slowly changing? Because if there is only one for the entire enterprise, it will change slowly, if at all.

Even if you happen to understand what the right model is, (and by model I mean data model, analysis model, process model, any model) and you manage to implement it while its still the right model, in a year its not going to be the right one. And a centralized, high cost, committed architecture won't and can't adapt. You'll still be paying the mortgage on the data warehouse.

Very large centralized models cannot be comprehensive and up to date, because to be comprehensive they have to be so complex as to be difficult to change, and as a result they quickly become out of date. It's sort of a Heisenberg uncertainty principle for common meta data repositories.

"Giving people their flying cars"

Madsen of course doesn't solve the entire problem in his keynote, but he points out some directions that make sense. And his graphic depicting a happy couple blasting off in their very locally controlled flying car sends the message- users can do their analysis without central oversight or interaction. (Although, one would imagine that some sort of air traffic control would be necessary, and the refueling stations for the cars would probably be run centrally- we're not advocating anarchy here.)

Having built data warehouses, established a data warehouse competency center, and provided business intelligence services for thousands of users, I can testify from first hand experience that centralizing alone is just not going to work. People who worked with me a decade ago will remember the significant amount of time spent creating meta data repositories. Are they still needed? Yes. But they simply can't do everything. Use them with care, and be wary of your ambition for them.

First, accept the fact that users are not mindless consumers. Learn from the fact that they use excel constantly, and they don't just read reports- they build things, adding data, fixing data, re-organizing data. They think. Give them tools that include them as part of the data processing.

Business intelligence cannot not be solely a process where formal requirements are gathered, followed by a publishing exercise of delivering the reports on time.

Are there some reports where this is the case? Sure. Monthly management reports and dashboards shouldn't change every month. The model can work for some amount of the delivered data analysis.

The entire architecture isn't getting ripped out- but if the new architecture is successful in bringing the pent up demand that is currently being satisfied by shadow systems into the light, then distributed, user centric, user driven business intelligence will become a significant percentage of the total.

But the old way of thinking has to change. Don't "Crack down on shadow systems".

Find a way to provide better service, be it self, assisted or centralized service that makes the shadow systems simply a less effective way to do it.

The existence of shadow systems, and the extent of them, is the clearest argument that centralized business intelligence alone is simply not up to the task.

Once you have people doing whatever they want in the self directed part of your architecture, DO watch what they are doing- not to control it, but to learn from it. Everyone constantly re-structuring the customer dimension? Obviously it's time for an update. By watching what users edit, what gaps they fill in, you can find the data quality issues, identify the fuel to put on the self directed fire.

Tools like Lyzasoft, our own Datamartist tool, and Microsoft's Power Pivot in Excel 2010 and others are all going to drive power to the users, and introduce a new balanced approach between centralized and local parts of business intelligence architectures. Visualization tools like Tableau will further give people the ability to create powerful, consumable analysis in a self serve mode.

Will there be challenges with data quality, risk management and wasted time doing pointless analysis? Most likely.

Will the information we gather and the payoff from the successful bottom up analysis efforts make it hugely valuable overall? I for one think so.

We need to learn to trust our colleagues with the data, while at the same time managing the reality of data quality and risk of errors that more free form techniques can create.

Companies that include both top down and bottom up capabilities in their architecture will stop wasting time fighting internally, and start to take advantage of all that data.

In the data analysis realm, spreadsheet errors are often used to try to dissuade companies from letting their people "work with the data directly". Software vendors of all sizes, but particularly the really big ones (those incumbants) spread FUD because if they can stop people from getting at the data themselves, it increases the chance of companies buying some more business intelligence suites.

The argument goes something like this:

Spreadsheets have been shown to be plagued with errors, many studies showing error rates above 90%. You need to reduce the risk that spreadsheets are creating in your organization by establishing formal, documented processes that are created an managed by professionals using sophisticated tools.

Then the usual nightmare scenarios are brought out, all involving rabid Auditors, Sarbane-Oxley, governance failures etc.

Now, don't get me wrong, spreadsheet errors are a very real and serious problem, and there are all sorts of data applications that should never be done in Excel or other ad-hoc, user driven tools. Ever. Formal documented processes are critically important, and there are lots of places where you better be using the right tools and professionals.

I have seen the culture of the spreadsheet completely undermine initiatives that would have driven better data quality, data analysis and business processes. The spreadsheet certainly has its dark side.

But the problem is that FUD paints with a broad brush. People take it as "Spreadsheets with data in them? Bad news. Don't do it. Individuals able to get at the data, and quickly transform it, analyze it? Who knows what they'll do- shut them down!"

Sadly, from a data quality point of view, sometimes the spreadsheets have the BEST data quality- because people have fixed the issues they can't fix in the transactional system due to constraints or IT department delays.

Encourage positive change with reasonable controls.

Intelligent, responsible people should be encouraged to use "informal" methods and tools to do data analysis.

These people will find things, learn things, and drive positive change (including change in those big formal professional systems).

They should do it with a reasonable understanding that doing things in an informal way, with spreadsheets or other tools does introduce errors, and should consider this when they recommend taking action based on the results.

Balance between two extremes

The totalitarian state: I don't think there is an IT department in the world that is capable of stopping all unofficial data analysis. In fact, I would suggest that the moment such an IT department comes into existence, it would kill the host company, a harsh sort of self-regulation. People interested in data and thinking for themselves would just pack up and leave. So who would be left making the decisions and based on what?

The twisted web of spreadsheets: Companies that allow an anything goes, visual basic code, macros and manual cut and paste direct to the annual report environment are not going to be long for the world either. They populate the horror story pages on the spreadsheet risk websites.

The zone of win.

You want to be somewhere between insane spreadsheet addiction and strict formal big tool paralysis.

I submit that companies that balance risk while still encouraging their smart people to "play" with the data and do analysis in new and interesting ways with new tools are going to win.

Again, don't let this process generate your profit and loss statement- understand where and what the informal discovery process is for- but do let it discover things. If it discovers something interesting you'll have the chance to check for the errors. Make sure its part of the process to do so.

By letting the FUD get you down, you'll never get that far and who knows what insights you might be giving up?

Of course, we believe you should go even further and give those intelligent, responsible people new tools that are less error prone than spreadsheets but still provide as much or even greater flexibility. That's why we're building Datamartist after all.

Openness, balance, and clear minded pragmatism will get you further than FUD every time.

I believe a key to understanding how information moves around your organization is to think of it as a mini economy. (I know, the economy is not our favorite subject right now, but bear with me).

There are information suppliers, and information consumers. The consumers are willing to pay more or less for different types of information, and different methods of supplying information have different costs. In the end, the market decides what gets done and what does not get done.

And like many markets, there is also a underground economy- places consumers go if the official prices don't make sense, or the products they want are not available on the open market.

In many companies, the IT department in theory has a monopoly on information supply, however the underground is active and constitutes a significant supply. The underground in this case is all the excel spreadsheets, the MS Access databases etc. used to make the shadow systems and spread marts. Spreadmarts seem to exist in the majority of enterprises- I've mentioned an interesting study regarding these shadow systems previously, and the attitudes people have.

To help illustrate this I am going to make up some data and put it in colorful graphs.

Looking at the first graph, in broad terms a data warehouse based approach will have higher costs than one based on data marts (because data warehouse provide more cross enterprise integration, which requires more effort), and the spreadmarts will have the lowest perceived cost. It's important to note that the actual cost of spreadmarts are higher, but percieved cost is what drives the consumers choice.

The trick is that because the percieved cost of spreadmarts is so low, and because there is no sanctioned enterprise solution to compete, a significant amount of effort is put in to these systems for any type of analysis that is percieved to be possible. Of course for certain data volumes or complexities there is no alternative to a full fledged data warehouse or data mart project, but for almost everything else, business users and analysts will often try to go it alone creating a chaos of spreadsheets and data bases.

The problem is, even "experts" can't accurately estimate how much effort the data analysis is. So estimates for how long it will take to "whip it up in excel" by non-experts are almost always low by orders of magnitude.

Don't dictate. Engage with sanctioned tools that work the way people want to work.

The key to adjusting this market imbalance is to introduce a new sanctioned product line, in effect undercutting the "black market".

This is exactly what self serve data transformation is about. Rather than leaving users to do it themselves in Excel- IT can provide specific tools, and thereby reduce the amount of completely opaque data transformation going on, while still providing users with the ability to get what they need.

So why is that better?

• It opens up the dialog - Talking is better than having a "Us" vs "Them" mentality. It lets you meet the people involved, lets you discuss their challenges with them, and provides an opening for discussion of important topics like data quality, master data management and data security.
• You'll know who the power users are - Right now, it is potentially anyone who has Excel- chances are that's everyone in your organisation.
• It gives you visibility on what matters to the business - If you know what the hot topics are, it can help you keep the official systems relevant and prioritize your efforts where they will do the most good.

What has to be different in this new relationship, however, is that IT has to understand about the "self" in self-serve. People will do things that no self-respecting ETL developer or data warehouse architect would ever sanction. If you clamp down and stop them, they will abandon the tools and return to the wild west. IT believes that it has the power in the relationship, but in fact the users are able to walk at any time. So add value, communicate, educate, but don't dictate. If your relationship with the business users, and the "Kings of the spreadmart" is poor to start, you have to give it time to evolve.

"But we just can't let them do that."

Resist the urge to clamp down.

Keep your systems secure, guard your infrastructure, but don't have any illusions that you can stop people from analyzing and transforming their data.

If they want to calculate net sales in a particular way then they'll do it in excel, and it will be the number that the CEO sees. The business is made up of grownups, after all. IT has a responsibility to explain the issues and challenges that shadow systems and rampant spreadsheeting can cause, but I have yet to see or hear of a company where an authoritarian approach works. As Princess Leia said- "The more you tighten your grip, Tarkin, the more star systems will slip through your fingers."

Arming the rebels

The business intelligence vendors are all realizing what the crowd pleasers are- really good integration into office applications, excel at the forefront. People want at their data.

Microsoft has of course long provided the main weapons for the shadow systems, MS Excel and MS Access- and they are going nuclear with the addition of "Power Pivot" to Excel 2010- although it is largely a presentation layer tool, and probably won't be used widely for data transformation itself.

Trying to fight all this with the standard tools of closing down the ability to export data, hiring an army of report writers, and constantly raving about the dangers and pitfalls of run away spreadsheets is like pushing on a rope.

Provide a safe, legal alternative to the free for all.

Talk to your business users. Understand their needs. Provide them with tools. Work with them to both empower responsible analysts, and avoid the worst issues that existing shadow systems are creating.

I believe in an approach that firmly places people in the center of a new way of looking at the data analysis process.

In the past, “big business intelligence” created an infrastructure heavy, highly centralised and technology focused approach to getting data from source systems into reports in the hands of the users. Under this regime, users were not to be trusted with raw data, but were given tightly controlled, managed and aggregated reports in order to protect the “single version of the truth”.

The theory and practice were tightly defined, and had been honed over decades of business intelligence and data warehouse orthodoxy. Giving raw data to end users would lead to chaos. Letting end users define new ways to look at the data would corrupt the master data, and lead to everyone looking at something different.

You can guess the sort of response this "don't give them the raw data" approach gets from capable, curious people that want to get down to some real analysis.

But to be fair you can see why these concerns are thought to be well founded. Almost every large enterprise is awash in a sea of excel files and a tangle of links and formulas. Excel is a wonderful tool, but it only offers the illusion of solving the data transformation problem. It is a much better reporting/dashboard tool than an ETL. (Although in the right hands it can do remarkable things.)

And this is the true state of affairs now. When the “official” system does not provide the answers that the business needs the people who need to make decisions get the data anyway, and they do it themselves. They do it in excel, they take night courses in Structured Query Language (SQL) they hire consultants (or even summer students) to build rogue data bases that they run on servers hidden under desks to get at the answers they need.

It is easy for the data warehouse theorists to highlight the clear issues with "spreadmarts" and "shadow systems".

But we need to be pragmatic. The reality of building a centralized structure that imposes strict formal rules and change management processes is that often while it does ensure that there is only one version of the truth, it is a version of the truth that no one can use because it has been so formalized, aggregated, compromised and delayed that by the time it is delivered the pressing business questions have changed and meaning has been expunged. The data warehouse becomes reporting rather than analysis.

Its clear that enterprises need this kind of reporting- I'm not advocating abandoning the existing approach- but augmenting it. Up till now, the solution has often been "more of the same".

The regime decided that the solution was to add more technology to the central systems, increase enforcement, and search out and repress all the dissident data manipulators. The data resistance was forced to go underground, to hide their spreadsheets, to outwardly appear to be following the official line.

It is very true that there are some risks in allowing people to analyze their own data, but there is also a reward. There are a small group of people who love data, who understand the business questions, who work to tease insight out of a steaming pile of raw data and can find things that are game changing. Massive, formal, designed by committee data warehouses can deliver a powerful and useful view of things, but they rarely offer flashes of insight. When they do, it is often during the design and discovery process- rarely by users using the system after it has gone live.

The Datamartist tool has been built based on the belief that both formal, centralized systems AND local, personal data transformation have a place in the architecture and that both should be official places.

People can be trusted with the data. In fact I think for an organisation to truly be successful at mastering its information, they have to be.

We have to realize that we can't allow our obsession with the quest for a single version of the truth to turn us into totalitarian regimes, certain that OUR truth is THE truth, and that messing around with the data is by its very nature subversive and dangerous.

Data to the people.

This is what I got when I searched for "business intelligence".

Apparently a lot of tweets containing "business intelligence" are microsoft focused. Can you even see the tiny "cognos", the "sap" in there? Oracle is readable- but wow. Microsoft. Sharepoint. Somebody in the Microsoft marketing camp needs to save this picture for when its time to talk about performance bonuses.

I've written here before about the Microsoft marketing machine, and when I did, I noticed a flurry of traffic to the Datamartist site from Redmond, and then noticed that I got some hits from twitter itself- Microsoft BI folks exchanging the link to the blog post. So the people selling and promoting Microsoft Business intelligence are definitely on twitter.

Whats also interesting, is to see the current marketing storms- guess what comes up when you search for "Microsoft", and guess what for "Google"? Yes, if you guessed "BING" for Microsoft and "Wave" for google you're right- but notice how "BING" made a double appearance- People talking about google are talking about bing, but people talking about wave aren't talking about Microsoft. I will leave it to the marketing experts to put forward their theories on all this.

This survey, published by the business application research center, reports that although your BI vendor might be telling you that baseline adoption is 20%, it turns out it's a lot less.

In any given BI-using organization, notes Nigel Pendse, a principal with BARC and the primary architect of BI Survey, just over 8 percent of employees are actually using BI tools. Even in industries that have aggressively adopted BI tools (e.g., wholesale, banking, and retail), usage barely exceeds 11 percent.

I have to admit that the around 10 percent didn't surprise me overall, but I was expecting that in those particular industries they would be much higher- i.e. even higher than the 20% figure that big BI has been telling us for years. Not having worked in retail or banking directly, I just assumed that they were all glued to their reports and cubes.

The article goes on to say that some indicators show that adoption is actually falling slightly. All this while the mega vendors have been focusing their message (and supposedly the capabilities of their tools) on "BI for the masses" and "self serve business intelligence".

So what does this mean? Well, maybe it means that all those people who are frustrated by their big business intelligence, are disappointed at how few people are actually using the reports, and are wondering how the other companies do it should know that, well, on the whole the other companies don't do it either.

On the other hand, maybe the "right" number of people using business intelligence is 5%. Maybe in some industries it's three people at head office and thats it. In other cases it might be that the target should be every single employee using BI every single day. I think in the end the ever useful "it depends" is in play here.

What I do believe, however, is that big business intelligence is broken (I'm talking about the mega vendors). If it's not completely broken, then it looks broken and everyone is talking about how it's broken. Which is just another form of being broken.

I wonder what would happen if we looked at the adoption rate for Microsoft Excel for data analysis? Going out on a limb here, but I'm betting it's more than 8%.

People vote with their behavior. Obviously, its not all doom and gloom, and it is possible to have a highly successful business intelligence deployment. In a former life, I had issues making sure we had enough funding for Cognos licenses because certain cubes were so popular that we had more demand than supply. But I've also looked at log files for entire data marts that were flat out empty. Hundreds of reports built and maintained without users.

What makes BI successful

Generally, the key difference wasn't the tool, or the features, it was the content. Data warehouses and data marts that had been driven by the business and were focused on what the business wanted to see, in the way they wanted to see it were successful. Projects that were technology focused, run by IT and did not have the needs of the end users anywhere on the map would end up doing nothing but consuming cash and data center space.

It simply does not matter if the report takes 2 seconds or 2 hours to generate if it is not important to the business and there is no decision to make after seeing it.

Big business intelligence seems to think that BI for the masses is a tool problem- something in how their portal works, or how many rows of data per second their appliance can process. Sure, if the tools are hard to use or learn, it's a factor, but I think more often than not business intelligence isn't used because it's not providing what is required.

There are a lot of very talented business intelligence professionals that work very hard to deliver the goods, and there are lots of very successful business intelligence projects of all sizes that create real value. But the whole industry needs to take a hard, honest look at where we've come from, where we are going, and be honest about who is using what and why.

Often, people use excel because last week they didn't know exactly what they needed, and it is a tool that lets them build it themselves this week when the boss wants the answer and there is a decision to make. With all its flaws, it's still the most adopted Business Intelligence tool in the world.

Should everything be done at lightning speed? Is it really never possible to know what analysis is needed and use a traditional business intelligence approach to creating it? Of course not.

But are there some types of analysis that need to be done very quickly? Yes. Should we as an industry ignore these, or "leave that to the users in excel". I don't think so.

I'm spending all my time building a self service data transformation tool because I believe there is room in the overall architecture for less formal and more creative, ad-hoc tools. Excel is the ultimate in informal (often far too informal). At the other end of the spectrum, a carefully run, tightly managed data warehouse project is the ultimate in formal (and often is overkill, hopelessly expensive and too slow for a rapidly changing environment).

I'm looking at the middle ground where we meet the users part way, and nurture capabilities to do rapid prototypes, one-time analysis and user driven data transformation. Not as a replacement for big BI- but as another tool in the tool box.

It is currently available for download here.

I'd very much like to thank all those who have signed up at datamartist.com over the past months to formally take part in the Beta- and thank you in advance for everyone's feedback and suggestions.

If you have signed up, you will have received an email with additional information, and the details regarding the Free Licenses that we'll be giving away to the most active, registered beta testers.

If you haven't signed up, but want to participate actively, please do so we can keep you up to date and provide more direct support.

Regardless of whether you signed up or not, your feedback is welcome.

There are still lots of powerful features to add (and no doubt bugs to kill), but I hope you'll be as excited as I am about what's possible with this tool. I encourage you to download it and give it a spin.

It is a quick download, and a simple install that will only take a minute or two. You'll find yourself manipulating data like you've never done before.

I've included a few sample data sets with the tool to mess around with, but of course it's most interesting with your own data.

I am convinced that the best source of new ideas and features will be from real users like yourself, so I'm working to ensure that it's easy to give feedback. I've embedded a feedback button directly in the application- this will let you fill out a web form (anonymously if you prefer). We always like to get mail as well, at support@datamartist.com.

Why do so many people turn to Excel for their analysis needs? Because with excel they are in control.  With Excel it IS personal. Massive, expensive data warehouses are one size fits none. Excel is the ultimate in customisation. 

“Despite years of investing in BI, many IT organisations have difficulty connecting BI with the business, and to get business users fully involved and out of the ‘Excel culture’”
- Bill Hostmann, Vice President and distinguished analyst at Gartner

Excel isn't always enough

Many users frustrated by the limitations in Excel in terms of data volume and functionality want more.  Some of them take up programming macros- others spend money they'd rather not on throw away database development.  But demand encourages supply, and new tools are beginning to emerge.

As Ted Cuzzillo put it so well in his trends for 2009 blog post recently:

“on business desktops, new, highly individualised tools will sprout.”

The Datamartist vision is that of a user driven tool that provides the flexibility of a spreadsheet with the key functionality of a database without the database. 

Personal data marts can for the first time be built directly on the desktop, without programming, yet handling millions of rows of data, and implementing truly useful data analysis.