Now, of course, standards are needed, and for a broad range of tools its counter productive (and horrifically expensive) to let everyone just use what they want. The cost of data centers, integration, etc. will be radically more if a company does not bring order to the chaos and the marginal advantages that a very specific or niche technology might have in one department is often obliterated by the increased support costs and integration issues globally.

But if a company looks at these savings that come from standardization, and extrapolates too far, they can fall off the other side of the benefits curve and find that they're hurting, not helping.

In the article in Harvard business review, the researchers also throw around the term "bitsmith" to describe someone who has both subject knowledge, and the ability to wrangle software, and to even create GASP! custom software that does what the team needs to get done. In many companies, current information technology dogma does not leave much room for people that have the time to be a "bitsmith".

In many companies "Custom software" is a four letter word. Well, I personally have used it, commissioned it and written it and know that often it can provide fantastic value- I've also seen people spend thousands of hours building something that never quite worked when an off the shelf tool a hundred times as good could be bought for a few thousand dollars. It's a matter of being realistic and learning how to see the difference between a problem that is specific to your industry/situation that could really benefit from some custom code, compared to a problem that is huge, main stream, and solved hundreds of times over by existing software vendors.

What I think can happen is a pendulum swing, where a company goes from "no standards, a jungle of wasteful custom software" to "Thou shalt use/buy only software on the following list."

The problem is, making a list that contains everything is just not possible. Things change. Stuff happens. It is possible that a single person might need a single piece of software that allows them to understand something, design something, communicate something that will make that software have a truly massive payback, justifying all sorts of pain and config on the part of technical resources and infrastructure.

The challenge, as always, is to have an open, working relationship between those entrusted with establishing and enforcing standards in terms of tools and those who are expected to use those tools to do business. As with so many things in governance, it's about balance, clear goals, and processes that allow for brilliance, change and creativity while not letting that process become the loop-hole that undermines all those savings standardization brings.

I believe in an approach that firmly places people in the center of a new way of looking at the data analysis process.

In the past, “big business intelligence” created an infrastructure heavy, highly centralised and technology focused approach to getting data from source systems into reports in the hands of the users. Under this regime, users were not to be trusted with raw data, but were given tightly controlled, managed and aggregated reports in order to protect the “single version of the truth”.

The theory and practice were tightly defined, and had been honed over decades of business intelligence and data warehouse orthodoxy. Giving raw data to end users would lead to chaos. Letting end users define new ways to look at the data would corrupt the master data, and lead to everyone looking at something different.

You can guess the sort of response this "don't give them the raw data" approach gets from capable, curious people that want to get down to some real analysis.

But to be fair you can see why these concerns are thought to be well founded. Almost every large enterprise is awash in a sea of excel files and a tangle of links and formulas. Excel is a wonderful tool, but it only offers the illusion of solving the data transformation problem. It is a much better reporting/dashboard tool than an ETL. (Although in the right hands it can do remarkable things.)

And this is the true state of affairs now. When the “official” system does not provide the answers that the business needs the people who need to make decisions get the data anyway, and they do it themselves. They do it in excel, they take night courses in Structured Query Language (SQL) they hire consultants (or even summer students) to build rogue data bases that they run on servers hidden under desks to get at the answers they need.

It is easy for the data warehouse theorists to highlight the clear issues with "spreadmarts" and "shadow systems".

But we need to be pragmatic. The reality of building a centralized structure that imposes strict formal rules and change management processes is that often while it does ensure that there is only one version of the truth, it is a version of the truth that no one can use because it has been so formalized, aggregated, compromised and delayed that by the time it is delivered the pressing business questions have changed and meaning has been expunged. The data warehouse becomes reporting rather than analysis.

Its clear that enterprises need this kind of reporting- I'm not advocating abandoning the existing approach- but augmenting it. Up till now, the solution has often been "more of the same".

The regime decided that the solution was to add more technology to the central systems, increase enforcement, and search out and repress all the dissident data manipulators. The data resistance was forced to go underground, to hide their spreadsheets, to outwardly appear to be following the official line.

It is very true that there are some risks in allowing people to analyze their own data, but there is also a reward. There are a small group of people who love data, who understand the business questions, who work to tease insight out of a steaming pile of raw data and can find things that are game changing. Massive, formal, designed by committee data warehouses can deliver a powerful and useful view of things, but they rarely offer flashes of insight. When they do, it is often during the design and discovery process- rarely by users using the system after it has gone live.

The Datamartist tool has been built based on the belief that both formal, centralized systems AND local, personal data transformation have a place in the architecture and that both should be official places.

People can be trusted with the data. In fact I think for an organisation to truly be successful at mastering its information, they have to be.

We have to realize that we can't allow our obsession with the quest for a single version of the truth to turn us into totalitarian regimes, certain that OUR truth is THE truth, and that messing around with the data is by its very nature subversive and dangerous.

Data to the people.

The first one talks to the question of what the perfect BI tool is- answer? One that people actually use. The second uses a story to illustrate that many uses of information to make better decisions are business intelligence- even if one of the Mega tool suites isn't used, and there is no massive data warehouse in sight.

Both of these blog posts remind me of my days in the business intelligence supply side of a large organization, where I was tasked with creating, maintaining and evolving the "official" business intelligence systems. Anytime I gave a presentation to end users (often a demonstration of a new tools capabilities and power) the same question would always come up- Can we export data into our spreadsheets?

Everytime I showed an export to Excel functionality, the release of tension in the room was noticable. Everyone relaxed. They could get their numbers out, they could take charge themselves, and could get the information they needed. We worked hard to try to provide the functionality in the core systems, but the reality is that the needs of the users changed, often more quickly than we could make our massive databases, ETLs and SQL structures evolve. We were beholden to our change management procedures, we had to analyze the impact of changes on the whole.

At times, it was suggested that we "crack down" on the proliferation of spreadsheets and home made MS Access databases that sprouted when a need couldn't be met by the main system. But the reality is, the best solution is to work with the analysts- provide them new tools, understand their needs and don't rule out solutions just because they do not follow the "standard" business intelligence methodology of servers, big integrated suites of tools and rigid IT department involvement.

As with all things in life, balance between the "proper" way to do things, and a pragmatic approach that values individual initiative and results is important. After all, it's the only way what is "proper" can ever evolve and improve.

So the idea of self serve business intelligence is a wonderful idea- the problem is that its not all about pretty graphs and fancy web user interfaces. You need to somehow design the data model so that every possible report that users dream up is possible- or move so much data modeling functionality into the report writer that it ends up looking more like a data transformation tool, yet is easy to use. There are "new" techniques that are getting lots of discussion- columnar databases are one, and certainly they provide interesting techniques, but only IF you've got meaningful data.

Here's something to try- call up your favorite Business Intelligence vendor, and ask for a demo. It will be wonderful, it will be clear, easy, and simple- and it will be done on a set of data that was made by someone who knew exactly what the demo was going to be.

Real world data is messy, and it often does not follow simple rules. As a result, it takes significant work to build analysis- and to have a system that is ready for ANY analysis that any user might think up at the moment is non-trivial.

But its a worthy pursuit. And I think there are three key fronts in this battle;

1. Reporting and Analysis front end tools
2. Data transformation and integration tools
3. Fixing the source systems.

All the big players have integrated suites of products that perform the functions of the first two categories- IBM/Cognos, Oracle, SAP.

There are also lots of very interesting new tools in the first category- Tableau is one that gets a lot of buzz. Of course, the king of graphing and dashboarding is still the spreadsheet, and Excel has the crown.

There are lots of tools for the IT department in the second category. Datamartist is a unique new tool for the end user in this area- a self-serve desktop tool for data transformation. It is a tool that allows users to quickly transform data sets to experiment and create new analysis that can then be queried and viewed using the tools in the first category.

But in the end, I think the number one limiter on achieving the dream of self-serve business intelligence will be getting a handle on the quality of the data in the source systems. Garbage in means garbage out- and the last two layers shouldn't have to tie themselves in knots trying to fix issues that are generated in the transactional systems.

But they do tie themselves in knots- and it is those knots that stop users from more freely and accessing their data using an intuitive and almost brainstorming approach- which in the end is the goal of self serve.

Unless the underlying data in the source systems is quality controlled, and designed to capture the information that is critical for analysts, then the Business Intelligence layer will have to work too hard to "fix" the data, keeping a large IT team busy writing code, and there's nothing self serve about that.

In the end, the data quality issue is often what makes data warehouses and data marts so expensive to build- and drives users to spreadsheets or even databases (microsoft access is a common one). This is what I call "self serve data transformation"- and this is what Datamartist does- if you're frustrated with the access you have to your data- download it and give it a try.

It is currently available for download here.

I'd very much like to thank all those who have signed up at datamartist.com over the past months to formally take part in the Beta- and thank you in advance for everyone's feedback and suggestions.

If you have signed up, you will have received an email with additional information, and the details regarding the Free Licenses that we'll be giving away to the most active, registered beta testers.

If you haven't signed up, but want to participate actively, please do so we can keep you up to date and provide more direct support.

Regardless of whether you signed up or not, your feedback is welcome.

There are still lots of powerful features to add (and no doubt bugs to kill), but I hope you'll be as excited as I am about what's possible with this tool. I encourage you to download it and give it a spin.

It is a quick download, and a simple install that will only take a minute or two. You'll find yourself manipulating data like you've never done before.

I've included a few sample data sets with the tool to mess around with, but of course it's most interesting with your own data.

I am convinced that the best source of new ideas and features will be from real users like yourself, so I'm working to ensure that it's easy to give feedback. I've embedded a feedback button directly in the application- this will let you fill out a web form (anonymously if you prefer). We always like to get mail as well, at support@datamartist.com.

As machines get faster and faster, new tools that use memory are going to create functionality and speed that just hasn't been possible before- and we will have to shift our concepts of what gets done where in the architecture.  The desktop is far from dead.

The Advantages of a Business Intelligence Workspace

Forrester Research calls these new tools business intelligence workspaces. 

The Datamartist tool that I am creating is an example of a powerful new BI Workspace. It runs on your desktop, takes full advantage of the dedicated hardware, and uses the lightning fast speed of RAM memory to provide performance that server based systems are hard pressed to touch when there are hundreds or thousands of users hitting them with demands.
On top of the speed advantages, a business intelligence workspace provides a level of control and flexibility that is impossible with large, multi user, production systems.
Borris Evelson of Forrester describes a business intelligence workspace as:

"a data exploration environment where a power user can analyze production, clean data with near complete freedom to modify data models, enrich data sets, and run the analysis whenever necessary, without much dependency on IT and production environment restrictions."

There has been a lot of discussion about memory resident data analysis tools.  Analysts get excited and focus on what the tools allow them to do- while IT and management often focus on the relationship between end users and the IT department, with concerns about the new power in analysts hands. 

As Mr. Evelson points out in the above quote, with a powerful tool on your desktop that lets you analyse millions of rows, make your own decisions regarding data model and content all in an environment without risk of impacting other users or the production systems, the role of IT might be diminished.  I look at these tools as a business intelligence sandbox- the question arises "is this just another way to create more spreadsheet chaos?"

In my opinion, the question to ask is- if we do not provide such powerful tools to analysts what is the alternative-  the alternative is where we are heading today- spreadsheet "He" double hockey sticks (to use a Canadian expression),  with escalating BI budgets not delivering sufficient value. 

Memory based analytics are not going to replace all the tools that came before, nor are they going to make the IT department a fifth wheel.  There is no magic fix.  But they will provide powerful capability right where it is needed most, and I believe will actually REDUCE the amount of data transformation being done in spreadsheets where it is truly hidden.

 I think that handled correctly, they can be an opportunity to bring Analysts and Technical folks together, not split them apart.

Reduce Business Intelligence Cost

-Business Intelligence workspaces satisfy needs that would otherwise have to be addressed by pricey custom developments that are used once and discarded

 -They help avoid heavy investments in new hardware/BI appliances that try to deal with performance issues that plague centralised systems

-They offer the possibility of real business value, because by lowering the cost of analysis both in terms of dollars and effort, they allow analysts to try more "what if" scenarios, more quickly

-If the analysts are dialed into the business, that means more insight, more actionable information- and more profits.

I don't think anyone realisticly thinks that excel is going away anytime soon- it will continue to be used for analysis, graphing and reporting.   But I think more and more data sets will be imported into excel after being created in a memory resident BI workspace that allows power users to REALLY get at their data, and do it in a way that is scalable, flexible, and robust.

And being from the IT department in my former life- I think that Business Intelligence workspaces with flexible and well thought out audit and collaboration features will one day bring much more order and value to the business intelligence world than the current tangle of spreadsheets can ever promise. 

With the right mix of tools, we can do all this without having to slow down the people who know the data, know what they need, and understand the business value of getting numbers NOW, to make decisions that matter.