Data integration can be like a really well put together pizza- lots of good cross-referencing cheese-data to keep everything in its place, great crust that supports it all, and a universal appeal that might even get people to try something they wouldn't normally consume (data wise).

But without data quality, data integration can make pizza that nobody really wants to eat, and rather than enhancing the value of your data, your data integration efforts can make your bad data even less consumable than it was on its own.

While combining data from multiple systems can generate huge insights, it is important to understand that moving it and combining it with data from other systems will not always increase its value.

With good quality data you can have fantastic results, but bad quality data requires so much effort and transformation that often your payback on doing the integration will be non-existent.

Data integration enthusiasm

So what happens when an enterprise hears its stomach rumble, and starts thinking data pizza?

Enthusiastic analysts spring into action, building various mockups of all the fantastic dashboards that they will be able to produce, once the data integration is done. Terms like "near-real time, balanced, cross-functional score cards" start to get bounced around, and pretty soon, budget proposals and appropriation requests are flying from color printers everywhere.

Whats unfortunate in many cases is that cooler heads don't stop to ask the question- "So... all this data we are going to put together, is it any good?"

When you are making your pizza, you have to know if the cheese has been left out a bit too long or the green pepper is soggy.

What can be worse, is that if heroic measures are taken to try to get the data to fit together, the integration jobs themselves might actually degrade the data quality further- or eliminate levels of detail that are not compatible, actually hiding important trends and structures. A risk of integrated dashboards is that they pander to the lowest common denominator.

So if you are planning to do some data integration, to build a data pizza, think twice about putting that moldy pepperoni from the CRM system on it- sometimes less is more.

In fact, it might be that data integration is not your first concern- improving the quality of the data in all those data silos will actually improve day to day operations immediately- and make any future data integration project cheaper, and more successful.

Any great chef will tell you- no matter how complex the recipe, and how impressive your kitchen and equipment, the raw ingredients matter.

The majority of the development cost in the current business intelligence methodology is often in getting the data out of source systems (Extract), and transforming it to make it consistent across all the various dimensions needed (Transform) and then putting it in a model that is easy to query and analyse (Load). The creation of these ETL jobs is made dramatically harder if the data in the source systems is not consistent.

Change is the challenge

Companies are not static- they grow, diversify, change strategies, reorganize, rename and restructure. They acquire other companies or are acquired. The structure and content of the data their systems often tells you this story, and if the proper work is not done to keep the data consistent with itself and the new situation then this story will be painful and complex.

Remember ten years ago when we acquired company X, but decided not to change their customer codes to our standard, so all the codes had an "X" prefixed so that we wouldn't have duplicates? Well, those X's are still there, and all our queries have to deal with multiple code structures.

Remember how we used to have three independent databases, one for each region, then when we went to the new data center and put everything into a single database, we ended up with multiple schemas and all those crazy views rather than consolidating into a single instance?

When the data migration project made the decision to reduce the project cost by not addressing data consistency, they simply pushed this cost in the future, most likely turning a one time expense into an ongoing and expanding annual business intelligence cost.

You end up with crazy ETL jobs that parse the same field in different ways depending on the date of the transaction, or on other fields- "If the transaction is before 2002, then the first digit of the product code means X, otherwise it means Y, unless of course its from the western division, who do it differently so then you need to look at field A and use the CASE statement..."

Reduce Business Intelligence cost through data cleanup

If your data is cleaner you'll reduce business intelligence cost across your entire BI architecture.

• Reduce ETL and report development cost- both initial, and the cost of ongoing maintenance. Every change request will take more time if all the models are complex due to underlying data complexity.
• Reduce hardware costs- complex queries require more processing, and bigger servers to meet that nightly load window
• Reduce time spent reconciling numbers. Complex ETL means that chances are business intelligence reports don't match up easily with the operational reports from the source systems. People will spend time constantly double checking these discrepancies, and it will undermine confidence in all data.

Fix the problem at the source. Not in the Business Intelligence.

Business intelligence is far too often left to fix all the issues in the source systems- and then becomes the focus of dissatisfaction when costs and delays become unacceptable.

I've heard people argue "Thats what ETL is for right? Why are you complaining?"

Assuming that the ETL will fix the sins of the source system is an inefficient and costly strategy.

Everything is a balance, perfection does not exist, but when deciding what to fix and what to leave, don't let a lazy data migration project saddle you with years of business intelligence costs- when it's time to bulk load data into the system, make it as right as you can.

Tableau has received a fair amount of positive response from the visualization world. Even Stephen Few, who isn't shy to point out when visualizations are straying from the straight and narrow has been supportive of Tableau from the start.

We're excited about this new access to such a great data visualization tool because we know that people who do visualization have to transform their data- and if users of Tableau want a flexible, visual data transformation tool, the Datamartist tool is an obvious choice.

Free Tableau vs Professional version

There are of course some limitations with the free version of Tableau, in comparison with the full featured professional version ($1600 USD per seat.)

• Data import capabilities - Only MS Access, Excel and Text files
• 100 000 row Limit per table
• 50 Mb Limit per organization for the web server.

None of these limitations will stop you from making some pretty fantastic visualizations as long as your final summarized data set fits within the limits and you put it in the right format. But it does mean that to use this version of Tableau, you need to use another tool to get those large data sets summarized. This is probably something you are doing anyway, because there is almost always some data cleanup to do.

Get your data sets ready- there are going to be some beautiful viz getting made

The trick with visualization of course, is that you need data to visualize. With tools like Tableau, as good as they are at making the pictures, you have to get the data set to them first- and we all know how many pre-formatted, all is well, no data quality issues data sets there are lying around the real world (Hint: none.).

But I'll tell you, once you do have the data, what a fantastic bit of interactive web based visualization Tableau can do.

Often, people use a combination of MS Excel and Access to create the datasets that they want, then connect to the Access database with Tableau. Of course, we suggest you try Datamartist instead. Datamartist and Tableau are a powerful combination- first, using Datamartist pull data from mutliple sources (Datamartist loads out of SQL Server, Oracle, MySQL, and at a license price much less than Tableau Professional).

Datamartist lets you join tables visually (using a venn diagram interface that we're proud of), segment data using rule sets, summarize millions of rows if needed, and generally parse and transform with an easy to use calculation engine. Once you have the data where you want it, export it easily to an Access database, and let Tableau Public generate the visualizations that you need.

I'm excited about Tableau's decision to make its power available in this public version. I intend to do some serious data crunching with the Datamartist Tool, followed by some interactive visualizations with Tableau Public. The beauty of the Tableau Public setup is that I can then publish the visualizations right here in the blog, and highlight what the combination of these two tools can do.

• $150 000 is the AVERAGE cost of business intelligence software for a DEPARTMENT
• ETL software (Extract transform and load) is also $150 000 on average.

And the rule of thumb for cost of effort and services is 5 times the software cost

I'm not making this up. Check the link.

In the end, Boris suggests that the cost of a single, fairly straight forward report might be $20,000. Of course as he rightly points out there are lots of variables, and it's a classic case of "it depends", but even so- clearly you want to be sure the reports add value when you are using a process that requires that kind of investment.

Boris mentions in passing that the cost of a single day of an external developer he uses for estimating is $800 USD. You can buy two licenses of Datamartist and take a friend out for dinner for that.

Don't get me wrong- for a number of applications you need the big enterprise stuff- but in my mind it makes sense to avoid it when you can. Enterprise business intelligence has its place, but there are alternatives. The rampant use of Excel spreadsheets is evidence of the fact there is huge demand for data out there. Try Datamartist and find another even more powerful way to get a the data for those cases where you need to do more than a spreadsheet, but it's not time to kick off a data warehouse project.

1) Establish scope clearly before you establish budget. This sounds obvious but its amazing how often I've seen this go wrong. There is no way to know what something will cost if you don't know what it is. If you leave the scope too broad, user expectations will drive feature and scope creep that will wipe out your original budget. There is only one way to have a clear scope- write it down. And the best kind of scope document is one that is signed by all the various players. Make sure everyone understands what is being delivered, when, and at what cost. If there are gray zones, at least flag them as potential areas where additional costs might arise.

2) Take a look at the data and talk to people who work with it. It seems obvious to say that what's in the data matters. (Or more accurately what's not in the data). But its amazing how many budgets are made without doing any serious analysis of the data that is actually in the source systems. Don't look at the data model, see a field called "customer birthday" and design functionality around it. Problems can range from missing data, to mandatory fields that are filled with garbage because "otherwise the system won't let us put the order through" to differences in interpretation of definitions between groups within the company. For example if all the Asian sales offices classify customers into the same segments, but have slightly different rules then you will need to "reallocate" this segmentation- even though it is the same field, and the same codes. That reallocation is an ETL job you didn't budget for, unless you found it in a pre-budget data audit. Often the key here is not to launch a massive data audit, but to find the people who have been trying to make the global reports in spreadsheets- they've run up against these sorts of issues, and can probably even offer some solutions. (That well respected analyst in head office who has already painstakingly established a cross mapping for for the customer segments working with his colleagues in Asia, for example). These same folks are also going to be key in terms of adoption of the final solution, since they are often the current source of data for the underground data system the new data warehouse is supposed to be improving on. By involving the key people, you gain credibility, save time, and ensure that the final solution addresses business needs.

3) Do proof of concepts for the tricky bits. In many projects there are areas where something is being tried for the first time (certainly in the more interesting ones)- not surprisingly this is often where the issues arise. One way to help quantify how much effort will really be required by these areas is to do some quick and dirty proof of concepts to validate the basic technical and/or functional aspects of the component or system. Often, if it is early in the project and software and hardware selection has not yet been done, your vendors will be willing to assist with a proof of concept (or even do it themselves) as part of the evaluation process. You can learn important things in this stage- For example, if you are doing a reporting project that needs to deliver 300 reports, by doing a proof of concept of 3-4 reports (even if they are not much more than mock-ups) you can at least get a first estimate of how much development effort is involved, how easy the tool is, and how the software performs. I've done proof of concepts that doubled my estimates- because when we actually sat down and used the tool, we realized there was additional data cleanup and hardware required to make it work. Better to know that before you set your budget rather than after. The ideal proof of concept is to actually build a "wire frame" version of the key data marts with real data and let the users try it out. Often, its possible to do this quickly, particularly with a tool that lets you do rapid prototypes of ETL transformations.

4) Involve the project team in the budgeting process As a project manager, the responsibility for the budget is ultimately yours, however by getting input from the experts in the various fields you can greatly improve its accuracy. Don't guess how long it will take to configure the server- go to the infrastructure team and find out how long it took the last three times they did a similar install and configuration. Be aware of the differences, but for many items by talking to the people who have been there and done that you will get good estimates of the true cost/duration of your project line items.

5) Watch out for Infrastructure and User centric costs The following costs are often overlooked, and end up being part of the cost overrun in the end. Don't get caught by these classic end of project costs;

• Infrastructure costs- We start to take our IT infrastructure for granted, and assume it will accommodate the new system without modifications- but is the network fast enough? Is there enough data storage? Just assuming that the existing server capacity, storage and bandwidth are sufficient may hide significant costs that the project will need to take on just to make the system operational. I've also seen cases where three projects that were launched at the same time all checked that the storage was available- but of course each project did not take into account the other two. The last project to go live ended up having to buy more hard drive capacity- and went over budget.
• Training for end users. There are few systems that don't require some amount of training for end users. Not taking this into account will provide a rude surprise at the end of the project. Costs here include actual training by third parties, but also travel expenses for trainers or trainees if they are not all based in the same location. Web based training can be a cost effective alternative, but results vary- and its difficult to ensure that "attendees" are really attending.
• Transitional support costs. If the project has a wide scope and involves a large number of users, be aware that there may be an initial spike in help desk calls and PC support as the system goes live. Depending on how your help desk is structured, you might end up paying more to your outsourcing company, or need to hire some temporary employees to help handle the extra calls for a few months.

6) Have a contingency amount included in your budget and do everything you can to keep it till the end There are two big mistakes commonly made regarding budget contingencies- first, not having one at all, and second, using it early in the project. Contingencies are often unpopular, first because by increasing the amount that needs to be approved, they might make it harder to get the green light for the project, and secondly because some view them as a "fudge" or a lack of willingness to do a proper cost estimate. However, ideally a contingency is a realistic number that should be based on the risk inherent in the project. Very small, simple projects might only need a 5% contingency. Large, complex projects that involve multiple departments, hundreds or thousands of users and multiple software and hardware vendors need higher contingencies. There are simply more things that can go wrong, and its not realistic to expect that cost analysis can be accurate enough to foresee everything. The key is not to consume your entire contingency with the first scope change- I've seen it happen again and again. Contingency is supposed to be for those unforeseen things- for example, a technical problem with the interaction between two vendors packages requires custom development to provide the original functionality envisioned, or requires more hardware than expected.

By spending the time required up front, a realistic, practical budget can be created- and you can get your project started off on the right foot.

This is what I got when I searched for "business intelligence".

Apparently a lot of tweets containing "business intelligence" are microsoft focused. Can you even see the tiny "cognos", the "sap" in there? Oracle is readable- but wow. Microsoft. Sharepoint. Somebody in the Microsoft marketing camp needs to save this picture for when its time to talk about performance bonuses.

I've written here before about the Microsoft marketing machine, and when I did, I noticed a flurry of traffic to the Datamartist site from Redmond, and then noticed that I got some hits from twitter itself- Microsoft BI folks exchanging the link to the blog post. So the people selling and promoting Microsoft Business intelligence are definitely on twitter.

Whats also interesting, is to see the current marketing storms- guess what comes up when you search for "Microsoft", and guess what for "Google"? Yes, if you guessed "BING" for Microsoft and "Wave" for google you're right- but notice how "BING" made a double appearance- People talking about google are talking about bing, but people talking about wave aren't talking about Microsoft. I will leave it to the marketing experts to put forward their theories on all this.

Here's the secret- If you answer yes to all of the following its probably a good idea to buy:

• You've configured your Enterprise Resource Planning (ERP) software exactly as the vendor who wrote the data warehouse or data mart expected you to.
• You use ALL the modules of the ERP- and do not have any "best of breed" applications for any areas.
• You are interested in the key performance indicators that they have included and they calculate them the same way your company does
• Your end users like the layout, formating and color choices in the reports included and have promised not to ask for changes.

In fact if all this is true, then you will probably save HUGE amounts of money by using a pre-built package.

But. Lots of things tend to get in the way of that dream. More often then not it's not the pre-built package that needs to be focused on- it's what you have in your system and how you've configured it.

ERP Customization

Anyone who has been involved in a large ERP project knows that there is always at least a bit of customization. If the ERP is a good fit to your industry, and the project is focused on avoiding customization and using all those "flex fields" and "category codes" as they were intended, then the pre-built data warehouse that is available for that ERP might be a very cost effective route.

But here's the rub; in many ERP projects rather than change the existing processes (which by the way is often the justification for installing an ERP- process improvement), the business users find reasons to "keep things as they are"- and as a result, the ERP bends to the existing process rather than the other way round.

This means that new tables are created to hold information that normally would be in a different format in one of the standard ERP tables. This means that that code in field "X" that normally would mean something is overloaded to mean three or four things. This means that field "Y" that isn't needed by your company ends up being used for something completely different.

Well, when the pre-built data mart plugs into your ERP, it doesn't know anything about those tables you've added, and its assuming that the data is in the standard tables. Its already configured its ETLs, and star schemas to use those two fields that you've high-jacked, and so all those reports and cubes don't make any sense without modifications.

To be fair, many of the pre-built data warehouse frameworks have evolved to have lots of flexibility and configurability in them- you can modify and map the ETL jobs provided to fit your data. You can turn off KPIs so that they don't load in false data.

But the trick is the more you have to modify, map and configure, the more effort you spend. The more you shut off, the less you have. In the end you can find that you are always trying to fit your specific requirements into the general, vanilla requirements that came out of the box. Often, this configuration is more effort and demands more compromises than just building what you need.

Non ERP transactional systems

In some ERP implementations, certain modules of the ERP are not used and instead a third party software provides the functionality. Often, key information is then moved in and out of the ERP as required for transactions via an enterprise application integration (EAI) tool. In many cases, the reason the additional system is used is because it has more functionality (and therefore stores more detailed and varied information) than the ERP module. As a result, the EAI tool cannot move all of the information into the ERP, only summaries and selected data. This means that for the functional area that is treated by that module at least some of the detail users are interested in does not exist in the ERP. And if it's not in the ERP, it won't be picked up by the pre-built solution.

Diverse and specific report requirements

Its amazing how the layout (column order, summarization, which key performance indicators where) of a report can be very very important to report consumers. Its also amazing how much work can be required to modify the hundreds of reports and cubes that came with the pre-built system. Report writers everywhere are nodding their heads. Enough said.

Data Quality Issues

Even if you've used all the tables and fields in the ERP exactly as needed, and the reports look just fine to your users, if they have not already been given access to all this information you will probably find that there is lots of data quality work to be done. This cost is there for both the build and the buy options- just be aware of it, because it means the buy option is going to be more than just the sticker price.

One ray of hope- very specific industry solutions

Hold on, the vendors say, but we have industry specific data warehouses and data marts- these are built just for your industry so they know your business. We have one for banking data marts, another one for pharmaceutical data marts, you can get a sales data warehouse specifically for retail, and so on.

Without a doubt the more focused the pre-built data warehouse or data mart is the more likely it is that its going to save you time and money.

Conclusion? Often when something sounds too good to be true...

The bottom line is, although you can tell I'm not head over heals for prebuilt datawarehousing, I think the field has evolved. But then again, so have the prices.

The key is to be realistic regarding how much modification will be required now, and going forward, and how many of those hundreds of cubes and reports that are included are actually going to be applicable to your business.

It is possible that a package exists that will provide the best value. Just be very very careful when you are comparing the costs- because its never quite as sweet a deal as it first seems.

Smaller, very focused pre-built data marts, particularly targeted to applications or modules that are used fully by an enterprise are probably much lower risk and more likely to provide real value than a massive "out of the box, install it in weeks not years" data warehouse.

My personal experience with pre-built products was that in the end, when the dust settled, we had touched almost every ETL job, and the reports that were the most used and useful were ones we had built from scratch, largely using data loaded out of custom ERP tables.

If you have customization in your ERP, you have to think hard about this. The reason the custom tables were put in there was because they hold information that was important enough to justify the development. This probably means that the information in them is going to be important to the business intelligence you are building.

The first one talks to the question of what the perfect BI tool is- answer? One that people actually use. The second uses a story to illustrate that many uses of information to make better decisions are business intelligence- even if one of the Mega tool suites isn't used, and there is no massive data warehouse in sight.

Both of these blog posts remind me of my days in the business intelligence supply side of a large organization, where I was tasked with creating, maintaining and evolving the "official" business intelligence systems. Anytime I gave a presentation to end users (often a demonstration of a new tools capabilities and power) the same question would always come up- Can we export data into our spreadsheets?

Everytime I showed an export to Excel functionality, the release of tension in the room was noticable. Everyone relaxed. They could get their numbers out, they could take charge themselves, and could get the information they needed. We worked hard to try to provide the functionality in the core systems, but the reality is that the needs of the users changed, often more quickly than we could make our massive databases, ETLs and SQL structures evolve. We were beholden to our change management procedures, we had to analyze the impact of changes on the whole.

At times, it was suggested that we "crack down" on the proliferation of spreadsheets and home made MS Access databases that sprouted when a need couldn't be met by the main system. But the reality is, the best solution is to work with the analysts- provide them new tools, understand their needs and don't rule out solutions just because they do not follow the "standard" business intelligence methodology of servers, big integrated suites of tools and rigid IT department involvement.

As with all things in life, balance between the "proper" way to do things, and a pragmatic approach that values individual initiative and results is important. After all, it's the only way what is "proper" can ever evolve and improve.

Although its true that in today’s economy, a lot of the multi-million dollar business intelligence projects are going to be cancelled or delayed, cheap data analysis is possible.

You just have to create cost effective data marts.

How can you reduce the cost of data analysis, cut your reporting costs, and avoid expensive business intelligence mega-projects?

Focus your efforts on Actionable Analysis

Before specifying a report or dashboard or data mart ask yourself “what action will I take based on what I see in this analysis, and how will that action move the business forward?”   If it’s not clear that some action can be taken based on the results, chances are there are better uses for the cash you’re planning on spending to get the report.  Target cost analysis first.  Reducing costs shows an immediate, verifiable return on investment.

Reduce the data scope as much as possible

Only analyze what you have to, and keep an eye on bang for your buck.  Don’t spend 80% of your budget cleaning up 5% of the data unless you really believe that data has something to teach you.  Eliminate dimensions from your data marts- really ask yourself if a dimension in the star schema is “nice to have” or critical.  Just because the data is there does not mean its worth the effort.

Reduce the report scope as much as possible

Don’t build reports no-one, or only few people use.   Consider providing an export function, so people can create their own reports in spreadsheets, rather than have hundreds of reports included in the scope of the project.

Use Desktop, Snapshot and One-time Analysis rather than full blown server based scheduled systems

There is a huge cost difference between a one-time analysis of a static data set, and a dynamic system that is able to load data daily or monthly from a transactional system automatically.  In many applications, it's the first analysis that gives you the insight.  Spending a lot of money to have the data mart refresh every day might not make sense if the first run gives you the majority of the information you need.

Obviously, its impossible to know where the insights really are- if we knew that we wouldn't have to do so much analysis.  But when resources are tight, you have to pick and choose.

Use the Datamartist Beta to create personal data marts on your desk

Sorry, I just couldn’t resist-  Datamartist is going to create a whole new price point for data marts.  If your pet business intelligence project has been delayed or canceled thanks to the economic meltdown, give Datamartist a try.  You’ll find its remarkable what can be done without programming, servers, or expensive consultants.  Once you've created a data set, you can export the information to your favorite data analysis or visualization tool and find those dollars to save.

Data Size Limits in Excel

There are different types of limits-

1. The size in rows and columns the actual spreadsheet has.
2. Excel's (and your PC's) ability to crunch the numbers in a reasonable time. (RAM, CPU)
3. The size of the files involved and load and save times.

In Excel 2003, a spreadsheet has rows 1 to 65 536 and columns A to IV. This makes it a grid 256 X 65536. In Excel 2007 the spreadsheet is much, much larger, with rows from 1 to 1 048 576 and columns from A to XFD. (Making a grid 16384 X 1 048 576). Now before you get too excited about how much space you have in 2007, the reality is that limits number 2 and 3 define how you can actually use that space. But it is more and more is good.

So lets kick the tires on large data sets in Excel 2007. For these very informal tests I'm using a Quad-core workstation with 4Gb of RAM, so the results I get represent a best case compared to a typical laptop or desktop PC. First of all- putting a million rows of data in Excel 2007 (even a "narrow table" of only 3-4 columns) slows everything down. Delete a column, and you'll often see a 5-10 second freeze-up while excel churns away in the background- roughly the same amount of time needed to save the file. Plus, when I push it I've had it lock up on me a few times- requiring some Ctrl-Alt-Del action to kill it. Even a narrow table such as this makes the Excel file be at minimum 15-20 megabytes. For the particular text file I used, the .txt version was 9 Mb, the .xlsx file was double the size at 18 Mb. I added a few columns and the file quickly became 80 Mb.

Also, strangely, doing exactly the same thing multiple times results in very different times to complete- when I'm mentioning times its the average of 2-3 trials (see graph).

 All in all, although Excel 2007 can technically store a million rows, I'd advise against it. There are other reasons its a pain- scroll bars and page-up page-down don't scale well to 1M rows- its just hard to copy 250000 rows accurately- takes for ever to get to the end, and then you overshoot by a mile, and page up again forever to find it etc. etc. (And yes you can use the Go To command on the Home>Editing>Find and Select>Go to - but a model of ease its not.

I can tell you, however, that using all the other features on more reasonable data sets (up to say, 100 k rows), I LOVE what it can do in terms of analysis and reporting. Once you have the data in reasonable result sets, there is no better place to have it than in Excel if you want full control in my opinion. But how to get it there. Next posts: how to link to data in Access and build a mini personal data mart. We'll learn how to make a personal data mart given the currently available tools. (And you just know there will be some posts later where I show you how to do the same thing, but using Datamartist. ) Update: Datamartist now available. Download the tool now, and find a whole new way to transform and managed your data, including managing huge data imports into excel.