So the idea of self serve business intelligence is a wonderful idea- the problem is that its not all about pretty graphs and fancy web user interfaces. You need to somehow design the data model so that every possible report that users dream up is possible- or move so much data modeling functionality into the report writer that it ends up looking more like a data transformation tool, yet is easy to use. There are "new" techniques that are getting lots of discussion- columnar databases are one, and certainly they provide interesting techniques, but only IF you've got meaningful data.

Here's something to try- call up your favorite Business Intelligence vendor, and ask for a demo. It will be wonderful, it will be clear, easy, and simple- and it will be done on a set of data that was made by someone who knew exactly what the demo was going to be.

Real world data is messy, and it often does not follow simple rules. As a result, it takes significant work to build analysis- and to have a system that is ready for ANY analysis that any user might think up at the moment is non-trivial.

But its a worthy pursuit. And I think there are three key fronts in this battle;

1. Reporting and Analysis front end tools
2. Data transformation and integration tools
3. Fixing the source systems.

All the big players have integrated suites of products that perform the functions of the first two categories- IBM/Cognos, Oracle, SAP.

There are also lots of very interesting new tools in the first category- Tableau is one that gets a lot of buzz. Of course, the king of graphing and dashboarding is still the spreadsheet, and Excel has the crown.

There are lots of tools for the IT department in the second category. Datamartist is a unique new tool for the end user in this area- a self-serve desktop tool for data transformation. It is a tool that allows users to quickly transform data sets to experiment and create new analysis that can then be queried and viewed using the tools in the first category.

But in the end, I think the number one limiter on achieving the dream of self-serve business intelligence will be getting a handle on the quality of the data in the source systems. Garbage in means garbage out- and the last two layers shouldn't have to tie themselves in knots trying to fix issues that are generated in the transactional systems.

But they do tie themselves in knots- and it is those knots that stop users from more freely and accessing their data using an intuitive and almost brainstorming approach- which in the end is the goal of self serve.

Unless the underlying data in the source systems is quality controlled, and designed to capture the information that is critical for analysts, then the Business Intelligence layer will have to work too hard to "fix" the data, keeping a large IT team busy writing code, and there's nothing self serve about that.

In the end, the data quality issue is often what makes data warehouses and data marts so expensive to build- and drives users to spreadsheets or even databases (microsoft access is a common one). This is what I call "self serve data transformation"- and this is what Datamartist does- if you're frustrated with the access you have to your data- download it and give it a try.

An Extract, Transform, Load (ETL) tool is an intermediate software application that extracts the data from the source system, transforms it (often another way to say it FIXES it) and then loads it into the destination system. They are also very expensive.

The destination system is usually a data warehouse or data mart, and most of the ETL tools available are server based.  The ETL tool and related development is key to any any data warehouse project (and represent a third or more of the cost on a typical project).

Although most ETL tools use a visual interface of one sort or another, at the core they require programming skills and specialized knowledge. Google "datastage training" and you'll see that there is an industry grown up around learning how to use these tools.

But there's nothing magical about it. If you ever made a spreadsheet with data from multiple sources, transformed the data, and then either made reports or moved the data into another spreadsheet then you have made (or most likely were an integral human part of) an ETL. The problem is that out of the box tools like Excel and Access are so flexible, that too much is possible.  Where to start?

The amazing thing is that EVERYONE needs ETL functionality, yet overwhelmingly the tools available are expensive, hard to learn and designed for the really, really heavy lifting.

Surely not every data manipulation task that is too much for Excel needs an enterprise ready server based ETL tool? Particularly in the current economic environment, oversized solutions are not an option.

A hard working analyst that has a bit of data analysis to do, and nothing but Excel or maybe Access on his/her desktop is short on options and long on messy spreadsheets or the need to "learn SQL in 21 easy steps".

The vision behind Datamartist is to provide an easy to use, powerful, yet low cost data transformation tool, that guides users to generate well structured data analysis sets.  And all at a price that represents less than a single day of those consultants you have to hire to use the other software you paid too much for.

This is the perfect time find out what easy, flexible, visual data transformation can be like-  download now.