Here's the secret- If you answer yes to all of the following its probably a good idea to buy:

• You've configured your Enterprise Resource Planning (ERP) software exactly as the vendor who wrote the data warehouse or data mart expected you to.
• You use ALL the modules of the ERP- and do not have any "best of breed" applications for any areas.
• You are interested in the key performance indicators that they have included and they calculate them the same way your company does
• Your end users like the layout, formating and color choices in the reports included and have promised not to ask for changes.

In fact if all this is true, then you will probably save HUGE amounts of money by using a pre-built package.

But. Lots of things tend to get in the way of that dream. More often then not it's not the pre-built package that needs to be focused on- it's what you have in your system and how you've configured it.

ERP Customization

Anyone who has been involved in a large ERP project knows that there is always at least a bit of customization. If the ERP is a good fit to your industry, and the project is focused on avoiding customization and using all those "flex fields" and "category codes" as they were intended, then the pre-built data warehouse that is available for that ERP might be a very cost effective route.

But here's the rub; in many ERP projects rather than change the existing processes (which by the way is often the justification for installing an ERP- process improvement), the business users find reasons to "keep things as they are"- and as a result, the ERP bends to the existing process rather than the other way round.

This means that new tables are created to hold information that normally would be in a different format in one of the standard ERP tables. This means that that code in field "X" that normally would mean something is overloaded to mean three or four things. This means that field "Y" that isn't needed by your company ends up being used for something completely different.

Well, when the pre-built data mart plugs into your ERP, it doesn't know anything about those tables you've added, and its assuming that the data is in the standard tables. Its already configured its ETLs, and star schemas to use those two fields that you've high-jacked, and so all those reports and cubes don't make any sense without modifications.

To be fair, many of the pre-built data warehouse frameworks have evolved to have lots of flexibility and configurability in them- you can modify and map the ETL jobs provided to fit your data. You can turn off KPIs so that they don't load in false data.

But the trick is the more you have to modify, map and configure, the more effort you spend. The more you shut off, the less you have. In the end you can find that you are always trying to fit your specific requirements into the general, vanilla requirements that came out of the box. Often, this configuration is more effort and demands more compromises than just building what you need.

Non ERP transactional systems

In some ERP implementations, certain modules of the ERP are not used and instead a third party software provides the functionality. Often, key information is then moved in and out of the ERP as required for transactions via an enterprise application integration (EAI) tool. In many cases, the reason the additional system is used is because it has more functionality (and therefore stores more detailed and varied information) than the ERP module. As a result, the EAI tool cannot move all of the information into the ERP, only summaries and selected data. This means that for the functional area that is treated by that module at least some of the detail users are interested in does not exist in the ERP. And if it's not in the ERP, it won't be picked up by the pre-built solution.

Diverse and specific report requirements

Its amazing how the layout (column order, summarization, which key performance indicators where) of a report can be very very important to report consumers. Its also amazing how much work can be required to modify the hundreds of reports and cubes that came with the pre-built system. Report writers everywhere are nodding their heads. Enough said.

Data Quality Issues

Even if you've used all the tables and fields in the ERP exactly as needed, and the reports look just fine to your users, if they have not already been given access to all this information you will probably find that there is lots of data quality work to be done. This cost is there for both the build and the buy options- just be aware of it, because it means the buy option is going to be more than just the sticker price.

One ray of hope- very specific industry solutions

Hold on, the vendors say, but we have industry specific data warehouses and data marts- these are built just for your industry so they know your business. We have one for banking data marts, another one for pharmaceutical data marts, you can get a sales data warehouse specifically for retail, and so on.

Without a doubt the more focused the pre-built data warehouse or data mart is the more likely it is that its going to save you time and money.

Conclusion? Often when something sounds too good to be true...

The bottom line is, although you can tell I'm not head over heals for prebuilt datawarehousing, I think the field has evolved. But then again, so have the prices.

The key is to be realistic regarding how much modification will be required now, and going forward, and how many of those hundreds of cubes and reports that are included are actually going to be applicable to your business.

It is possible that a package exists that will provide the best value. Just be very very careful when you are comparing the costs- because its never quite as sweet a deal as it first seems.

Smaller, very focused pre-built data marts, particularly targeted to applications or modules that are used fully by an enterprise are probably much lower risk and more likely to provide real value than a massive "out of the box, install it in weeks not years" data warehouse.

My personal experience with pre-built products was that in the end, when the dust settled, we had touched almost every ETL job, and the reports that were the most used and useful were ones we had built from scratch, largely using data loaded out of custom ERP tables.

If you have customization in your ERP, you have to think hard about this. The reason the custom tables were put in there was because they hold information that was important enough to justify the development. This probably means that the information in them is going to be important to the business intelligence you are building.

With data marts (just like leasing a car) you get behind the wheel quickly, and it gets you where you want to go in style.  And the monthly payment is something you can afford now.  However, long term, well, in three years you don't own it, and have paid a bundle.

But let's be realistic.  Just as having all the cash on hand to buy the car outright just might not be in the cards,  a true data warehouse might require a very significant outlay before anything comes out the other end, making it unaffordable.  A quick, focused departmental data mart could be delivering value in a matter of weeks with relatively little investment.  (Your actual mileage may vary- depending on where you're at, its always dangerous to believe someone when they say "a matter of weeks" when software and people are involved.)

Will that departmental data mart, or even a number of data marts lead you to a single version of the truth?  Will it give you deep competitive advantage through a culture of data analytics and cross enterprise master data management? In my honest opinion, No.

But is it something you can afford in today's economy, and will you learn things about your data, your company's information culture, and your business that will be useful if in the future you embark on a true data warehouse initiative.  Yes.  Yes it is, and yes you will.

And I'll take it one (blatantly promotional) step further.  Is a personal data mart on your desk top as good as a full fledged departmental data mart with an army of highly paid developers maintaining it?  Probably not.

Is the personal data mart on your desk basicly free in comparision to the servers, software and hired help the data mart requires?- Yes. And does it, just like the data mart does for the data warehouse, prepare the ground for the next evolution when the economy turns around? Yes. Yes it does.

In difficult times companies that are pragmatic, and do what is possible, preparing for the day when more will be, survive to see that day.

It seems obvious that doing nothing because you can't afford to do the best thing is a bad strategy- but we need to ask ourselves, how often do we make that exact choice through inaction?

Data warehouse or Data Mart?

Data Warehouse:

• Holds multiple subject areas
• Holds very detailed information
• Works to integrate all data sources
• Does not necessarily use a dimensional model but feeds dimensional models.

Data Mart

• Often holds only one subject area- for example, Finance, or Sales
• May hold more summarised data (although many hold full detail)
• Concentrates on integrating information from a given subject area or set of source systems
• Is built focused on a dimensional model using a star schema.

More Info about Data mart models

• Data mart Modelling 101
• Data mart example
• Dimension tables and Fact Tables

More Detail regarding Data Warehouse Vs Datamart: and Inmon vs Kimball

As the concept of decisional systems, and data warehouses and data marts evolved, two major points of view came into existence. There are two giants in this field. Bill Inmon, and Ralph Kimball.

There are some that argue the best approach is to start with data marts, department by department, then merge them together to form a data warehouse- this is more in line with Kimballs approach.

Now, Bill Inmon is an advocate of the Data warehouse.  Here's one of his articles, which contains the following quote that makes it clear what he thinks about the idea:

"You can catch all the minnows in the ocean and stack them together and they still do not make a whale," Bill Inmon, January 8, 1998.

Ralph Kimball, on the other hand, advocates what he calls a bus architecture data warehouse.  His methodology specifies conformed dimensions, where multiple fact tables share common dimensional tables.  For me, each of these fact tables represents a data mart.  The row of dimensional tables that all the fact tables plug into is the bus,  and because, for example, the finance and the sales data marts both use the same product dimension table there is integration between departments.

The more extreme data mart strategy is that of the completely stand alone data mart, the concept being that its fast, easy, cheap, and delivers value immediately. I'm a supporter of this at the desktop level- thats the point of the Datamartist tool afterall.  But I don't buy this for server based architectures- what is really fast, easy and cheap when you have to buy servers, create a project and form a commitee?  In my mind if you've decided you need  a central server solution, some level of integration is needed, and don't pretend its going to be magic. 

The interesting thing about these approaches, is that the harder you work on really conforming your dimensions, the more your data marts look like the data warehouse that Inmon advocates. (Data modellers in the know will be jumping up and down right now shouting NO they don't- but this is a high level conversation...)  But the reality is, even in a data warehouse, issues will arise that require compromise- things that just don't map or conform, and budget, schedule and business reality will mean that nothing is ever perfect, and in the end the world is full of data warehouses that are less conformed than some data mart clusters. Its just not simple.

Generally, it is probably true that data warehouses provide a solution that is closer to the "single version of the truth", but they do take a HUGE amount of effort, and an ability to coordinate across the entire organisation. If you have not already built at least half a dozen data marts, don't think you can estimate how much effort a data warehouse will take. You can't. And bring your cheque book.

Whereas data marts might deliver some value early, if built without sufficient effort on cross functional mapping and data cleanup they are just more silo systems and have their own set of costs and issues. Don't measure payback on datamarts in years-  nothing is the same in a few years, you'll be back to the drawing board shortly. 

It's a real dilemma.  So which one? Data warehouse?  Data mart?  In my view, the right answer is "it depends" and "yes". However, never launch a data warehouse project as your first shot.  A successful strategy will balance the fast, pain point addressing solutions, with a medium and long term plan, and investment in infrastructure and competencies to build the technology, processes and culture that a company needs to manage information. Depending on your industry and how sucessful you are, a massive data warehouse might be in your future.  But sorry, no magic bullet.

Build a multi-level data strategy

• Level 1- Get the data to the people
• Level 2- Build Departmental Data Marts
• Level 3- Plan long term infrastructure and architecture

Don't do these things in order- this isn't step 1, 2, 3-  actively work on all three levels at once and ensure the plans at each level are coordinated.

Data to the People

People are building spreadsheets, and spending money on data base development now- you know they are.  Give them better tools, help them better use the spreadsheets, and formalize the way they do.  The do-it-yourself exists- but it doesn't have to be completely informal.

The Datamartist tool is adding another capability that can accelerate the process-  letting you move more quickly, proto-typing and analysing to determine which areas are ready for additional analysis capability. 

 In some cases Datamartist itself might simply be the best choice for certain types of analysis, cutting costs dramatically. In addition, if your end users are building their own data marts, when it comes time to build server based data marts they know the concept, they understand the structure, and can even provide concrete examples of the dimensions they need.

But whatever you do- don't make the mistake of thinking this is all you need.  Work on all levels at once.

Build Departmental Data marts

If a whole department is flying blind, and big money is on the table, then don't launch a three year data warehouse project- create some departmental data marts. These projects should be designed to be 3-6 months long, and be sold to management honestly and clearly as being for short term gains, and as part of a broader discovery process.  The hardware and software licenses will be reusable- but be clear that the data marts will have a limited lifespan- they always do.

And trust me, when you build these data marts you will discover all sorts of things about your data, your organisation, and your definitions and business processes.  You will discover that the sales organisation needs to analyse product segments in a way that is fundamentally inconsistent with how finance has been reporting it for years- and neither group is willing to budge (and they may both be right).  You will discover that 80% of your sales orders have errors on them or are incomplete.  These discoveries will help you build the next data mart, and assess if a data warehouse is possible.  They should also send you back to your transactional system and business processes to work to clean up the problems.

Build the infrastructure and deal with the foundation

There are lots of pitfalls in creating a decisional architecture-  this short list of from Gartner resonates with me- I've seen and battled these issues on project after project.

Set standards in terms of tools, project management etc.  Buy infrastructure for multiple projects, not on a project by project basis- don't have multiple servers when one with multiple data marts is radically cheaper.

And probably most important, I honestly think that you can build anything you want, any way you want, but it will not succeed if you don't have your definitions, both data and information, under control. (See Gartner issue #8)  In the end, it's not what language you speak, it's if you have a dictionary or not, and if everyone is using the same dictionary across your organisation.

You should have common reference data sets that are used by all levels. Datamartist can load in and use reference data that is coordinated with departmental data marts and the eventual warehouse. Make these data sets available to everyone- you'll be amazed that if they are easy to get and use, people will put them in their spreadsheets, and things might actually start matching up.