The majority of the development cost in the current business intelligence methodology is often in getting the data out of source systems (Extract), and transforming it to make it consistent across all the various dimensions needed (Transform) and then putting it in a model that is easy to query and analyse (Load). The creation of these ETL jobs is made dramatically harder if the data in the source systems is not consistent.

Change is the challenge

Companies are not static- they grow, diversify, change strategies, reorganize, rename and restructure. They acquire other companies or are acquired. The structure and content of the data their systems often tells you this story, and if the proper work is not done to keep the data consistent with itself and the new situation then this story will be painful and complex.

Remember ten years ago when we acquired company X, but decided not to change their customer codes to our standard, so all the codes had an "X" prefixed so that we wouldn't have duplicates? Well, those X's are still there, and all our queries have to deal with multiple code structures.

Remember how we used to have three independent databases, one for each region, then when we went to the new data center and put everything into a single database, we ended up with multiple schemas and all those crazy views rather than consolidating into a single instance?

When the data migration project made the decision to reduce the project cost by not addressing data consistency, they simply pushed this cost in the future, most likely turning a one time expense into an ongoing and expanding annual business intelligence cost.

You end up with crazy ETL jobs that parse the same field in different ways depending on the date of the transaction, or on other fields- "If the transaction is before 2002, then the first digit of the product code means X, otherwise it means Y, unless of course its from the western division, who do it differently so then you need to look at field A and use the CASE statement..."

Reduce Business Intelligence cost through data cleanup

If your data is cleaner you'll reduce business intelligence cost across your entire BI architecture.

• Reduce ETL and report development cost- both initial, and the cost of ongoing maintenance. Every change request will take more time if all the models are complex due to underlying data complexity.
• Reduce hardware costs- complex queries require more processing, and bigger servers to meet that nightly load window
• Reduce time spent reconciling numbers. Complex ETL means that chances are business intelligence reports don't match up easily with the operational reports from the source systems. People will spend time constantly double checking these discrepancies, and it will undermine confidence in all data.

Fix the problem at the source. Not in the Business Intelligence.

Business intelligence is far too often left to fix all the issues in the source systems- and then becomes the focus of dissatisfaction when costs and delays become unacceptable.

I've heard people argue "Thats what ETL is for right? Why are you complaining?"

Assuming that the ETL will fix the sins of the source system is an inefficient and costly strategy.

Everything is a balance, perfection does not exist, but when deciding what to fix and what to leave, don't let a lazy data migration project saddle you with years of business intelligence costs- when it's time to bulk load data into the system, make it as right as you can.

Knowing this, it's not surprising that a huge amount of data collection, extraction, and transformation happens in Excel spreadsheets, or Access databases that are made without the involvement (and often under the direct scorn of) the IT department in large companies. In my previous life I was in the IT department, and I saw some amazing systems generated with hundreds of spreadsheets and databases. This mix of spreadsheets and databases, created without the involvement of the IT department by power users or external consultants (financed out of departmental budgets) are often referred to as Spreadmarts or Shadow Systems.

For an interesting survey on the subject, take a look at TDWI's report "Strategies for Managing Spreadmarts: Migrating to a Managed BI Environment". This report is now a year old, but I'm certain as valid as ever.

The title suggests that the solution is managed BI- I won't get into that right now, but you'll notice the study was sponsored by the likes of Microsoft, Cognos, Microstrategy and SAP- so of course the solution is Big Business Intelligence solutions.

But what's really interesting from the survey, is how the different groups within the respondent companies feel about spreadmarts and shadow data systems. The analysts love them, the executives are unsure, and IT hates with a passion. This makes for an interesting mix.

This is very much what I've seen in my experience. IT and the Business are at odds with each other, and senior management is either disinterested or forced to take sides.

Where do I stand? I'm in the "avoid them if you can" camp when we're talking about a tangle of spreadsheets and undocumented MS Access databases that can be error prone and time consuming. I understand why it's often unavoidable, but I've seen first hand how painful these systems are to maintain.

On the other hand, I don't subscribe to the school of thought that says "Excel needs to be eliminated- analysts should use the Business Intelligence systems only, otherwise there will be chaos." Let's not go overboard. Excel and spreadsheets are useful tools, and have their place. Additionally, I really feel for business users who simply can't get what they want from the IT departments. I used to be the IT department, and it was frustrating to not have the resources available to build what people needed.

As one of the authors of the above report, Rick Sherman, said in a recent podcast:

"reality is no matter how many IT folks that you have in your company you're not likely to have enough resources or time to meet every business users reporting or analytical requirements..."

He presents what is a refreshingly balanced approach to Excel. In his predictions for trends in 2009, number 5 is "Excel becomes an accepted tool in a BI portfolio". He points out that this may not be mainstream in 2009, but I hope he's right about the trend. A pragmatic, inclusive strategy with more power to the people while avoiding the chaotic side of spreadmarts is where the solution is.

Check out the first four parts of this series (1,2 , 3 and 4) where we created an example data mart, with some fictitious purchasing data.

The final step is to join the dimensions we have created to the fact table. To do this, we connect up the two dimensions (Vendor and Item) to the Join block and connect an export block to the output. What has in effect been created is a complete Extract, Transform Load (ETL) and the final star schema join.

(If thats a bit hard to read- click on the image to see the full size screen shot.)

With the generated data set I used for this example, summarizing the data to yearly totals but keeping all the detail on Vendor and Item causes the roughly 4 million row raw data file to be reduced to around 800 thousand rows. (This summarizing was done on another canvas- although it could have been done on this canvas just as easily).

This data mart, with 800 k rows and two dimensions of about three thousand members each took my laptop about a minute and 45 seconds to solve, and save to a 360 Mb text file out.

Of course, by summarizing or filtering (just add blocks) analysis subsets could easily be exported directly to Excel, managing the data volumes involved, and letting you create the graphs, dashboards and reports that you need.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Now we need to create a Vendor dimension table and join it to this fact table to determine who our big vendors are.

In Datamartist it is a simple task to create this vendor dimension. As always we use blocks and connect them together. We define a dimension by using a reference definition block. All we have to do to configure the reference block is to specify which columns uniquely define the dimension (or almost uniquely, Datamartist will resolve duplicate keys using a majority/first rule set for you if you have some data glitches).

We start with an import block that brings in the Vendor master text file, then we define the reference by specifying "Vendor_ID" as the key. These first two blocks look like this:

Then we join it to the fact table we created in part two of this series with a join block. This means that now instead of just the vendor ID number that was in the fact table, we have the name, and address for the vendor in our mini star schema.

And finally we put a summarize block after that to total up all the monthly values for each vendor, and we export to excel. This is what the canvas looks like:

After we do this, we grab the excel file Datamartist just created for us, do a quick sort, and come up with a list of Acme's top ten suppliers. Feeling pretty good about ourselves, we do a review with the head of purchasing.

"Where's Mega brothers?" she says with a frown "I think your data is screwy- no way that Mega brothers didn't make the top ten- we spend a fortune on railways, and a lot of our freight goes with the Mega Brothers Rail company. Of course it is probably entered under different vendors, each location works with the office local to them... But we've got to view them as a single vendor in the data mart- you can do that right?"

Fixing Duplicate Rows

Having to deal with duplicate data is a very common issue in any type of data analysis. So, back to the canvas. By simply adding a de-duplicate block to our Vendor dimension table (after the Reference block, and before the join) we can find and resolve the Mega Brothers duplicates.
We just use the filter to find the records- (Easy to do, looking for "Mega" "rail" "brothers" etc. and we map them to a single instance.) This is the filter control that lets us find and tag the duplicates:

As we tag them, they show up in the mapper, which lets us see which duplicate records we have eliminated for the dimension. We run the canvas again, and this time, sure enough, Mega Brothers Rail is in our top ten. But even though the head of purchasing knew it was a lot, this is actually the first time she's seen the number. "Wow. I've got to give them a call- can you give me that in an Excel spreadsheet?"

Stay tuned, more to come as we go further into Datamartist's ability to segment, filter and organize large data sets.

If you want to see the interface in action watch our first Tutorial Video. Or just get right to it with your own data- download the free trial now- there is no registration required, and it installs in minutes.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

With data marts (just like leasing a car) you get behind the wheel quickly, and it gets you where you want to go in style.  And the monthly payment is something you can afford now.  However, long term, well, in three years you don't own it, and have paid a bundle.

But let's be realistic.  Just as having all the cash on hand to buy the car outright just might not be in the cards,  a true data warehouse might require a very significant outlay before anything comes out the other end, making it unaffordable.  A quick, focused departmental data mart could be delivering value in a matter of weeks with relatively little investment.  (Your actual mileage may vary- depending on where you're at, its always dangerous to believe someone when they say "a matter of weeks" when software and people are involved.)

Will that departmental data mart, or even a number of data marts lead you to a single version of the truth?  Will it give you deep competitive advantage through a culture of data analytics and cross enterprise master data management? In my honest opinion, No.

But is it something you can afford in today's economy, and will you learn things about your data, your company's information culture, and your business that will be useful if in the future you embark on a true data warehouse initiative.  Yes.  Yes it is, and yes you will.

And I'll take it one (blatantly promotional) step further.  Is a personal data mart on your desk top as good as a full fledged departmental data mart with an army of highly paid developers maintaining it?  Probably not.

Is the personal data mart on your desk basicly free in comparision to the servers, software and hired help the data mart requires?- Yes. And does it, just like the data mart does for the data warehouse, prepare the ground for the next evolution when the economy turns around? Yes. Yes it does.

In difficult times companies that are pragmatic, and do what is possible, preparing for the day when more will be, survive to see that day.

It seems obvious that doing nothing because you can't afford to do the best thing is a bad strategy- but we need to ask ourselves, how often do we make that exact choice through inaction?

Although its true that in today’s economy, a lot of the multi-million dollar business intelligence projects are going to be cancelled or delayed, cheap data analysis is possible.

You just have to create cost effective data marts.

How can you reduce the cost of data analysis, cut your reporting costs, and avoid expensive business intelligence mega-projects?

Focus your efforts on Actionable Analysis

Before specifying a report or dashboard or data mart ask yourself “what action will I take based on what I see in this analysis, and how will that action move the business forward?”   If it’s not clear that some action can be taken based on the results, chances are there are better uses for the cash you’re planning on spending to get the report.  Target cost analysis first.  Reducing costs shows an immediate, verifiable return on investment.

Reduce the data scope as much as possible

Only analyze what you have to, and keep an eye on bang for your buck.  Don’t spend 80% of your budget cleaning up 5% of the data unless you really believe that data has something to teach you.  Eliminate dimensions from your data marts- really ask yourself if a dimension in the star schema is “nice to have” or critical.  Just because the data is there does not mean its worth the effort.

Reduce the report scope as much as possible

Don’t build reports no-one, or only few people use.   Consider providing an export function, so people can create their own reports in spreadsheets, rather than have hundreds of reports included in the scope of the project.

Use Desktop, Snapshot and One-time Analysis rather than full blown server based scheduled systems

There is a huge cost difference between a one-time analysis of a static data set, and a dynamic system that is able to load data daily or monthly from a transactional system automatically.  In many applications, it's the first analysis that gives you the insight.  Spending a lot of money to have the data mart refresh every day might not make sense if the first run gives you the majority of the information you need.

Obviously, its impossible to know where the insights really are- if we knew that we wouldn't have to do so much analysis.  But when resources are tight, you have to pick and choose.

Use the Datamartist Beta to create personal data marts on your desk

Sorry, I just couldn’t resist-  Datamartist is going to create a whole new price point for data marts.  If your pet business intelligence project has been delayed or canceled thanks to the economic meltdown, give Datamartist a try.  You’ll find its remarkable what can be done without programming, servers, or expensive consultants.  Once you've created a data set, you can export the information to your favorite data analysis or visualization tool and find those dollars to save.