The transactional systems that a company uses can be the glue that binds, or can be a key part of the walls that block inter-departmental collaboration and information sharing.

The data migration project is your best chance to break down the data silos- but it won't be the easy path.

Not only can the ERP project create new opportunities for efficiency and process improvement, but it can also be the process through which people from across the company start to work together.

ERP projects often require teams of subject matter experts from various functional areas (finance, sales, manufacturing) to work together- depending on how serious your silos are it might be the first time many of these people have met.
Make the most of it- see if you can't encourage some new working relationships that last beyond the project. A few key personal contacts between departments can make a huge difference.

Best case and worst case: big difference

1. Best case- processes that stumbled along without any coordination are totally reworked for the better and everyone- including your customers- see a huge positive difference that ends up hitting your bottom line. Win!
2. Worse case- no-one will accept change, everyones position is that either things stay the same, or others adapt to their vision of the future- you end up finding a way to shoehorn all the existing processes into the ERP, resulting in a messy, customized compromise that no one likes and limits progress while costing a fortune. So sad.

If you are doing a data migration project- how can you help make it be a more positive and unifying step, rather than a transfer of the existing silos intact into the new ERP at great expense?

The first step is to admit you have a problem.

The first thing to do is to identify how bad your silos are, and get all the players to look the problem in the face.
If you have silos, but people don't identify it as an issue, there is no way you'll get the resources you need to fix them.

Make it clear that customization is the enemy.

Sure you could customize the ERP application that you are installing to do exactly what each department does now. Add some tables, write some bolt on code.

I'm sure you can do it. Thats not the point.

Technical resources love customization because it's more fun than configuration. Don't fall into the trap of thinking "we're satisfying the customers requirements."

Customization in an ERP system is expensive, usually reduces functionality and in the long term drives significant maintenance and upgrade costs.

If you customize the new ERP system to accommodate the existing silos you are NOT satisfying requirements. You are leading the business towards a failure that they don't really understand.

You need to find language that the functional teams and management can understand and explain the risks clearly.

If we don't get together and consolidate our processes and our data definitions we're going to end up with a mess in the ERP.

The system was not designed to do that- the point of an ERP is to be integrated- if our departments don't work together, than the ERP isn't going to make anything better- and its a waste of money.

Get top down support for the painful change that is necessary.

Sometimes things can be grassroots, originating at a level below the executive suite. If that works for your company, thats great.

But in the majority of cases the kind of short term pain that breaking down entrenched silos will cause is too intense to be initiated anywhere but at the very top.

The leadership needs to make it clear that everyone is going to share in the change, and no-one has a "get out of change free" card.

It has to be clear that it is not a competition between the various approaches that might exist, but a move to a new approach.

It's really really hard. If it's not hard, you're not doing it right.

There's nothing I can write here to make this kind of change easy, it's not.

But by building cross functional teams, making goals clear, and managing expectations it's possible to make progress.

So in summary what I've been trying to convey in this series of posts:

• Data migration projects are data quality projects.
• Data migration projects are master data management projects.
• Data migration projects are an opportunity to break down the walls between silos within the organisation.

You can make a real difference if you strive to make the data migration project advance on these three fronts.

You always have to work within the constraints you have in terms of budget, company culture, existing systems and of course, internal politics, but if you take a step by step and pragmatic approach with these goals in mind, you'll be contributing to the solution.

]]> http://www.datamartist.com/data-migration-part-5-breaking-down-the-information-silos/feed 0 http://www.datamartist.com/build-vs-buy-the-pros-and-cons-of-premade-data-warehouses-and-data-marts http://www.datamartist.com/build-vs-buy-the-pros-and-cons-of-premade-data-warehouses-and-data-marts#comments Thu, 28 May 2009 03:10:00 +0000 James Standen http://www.datamartist.com/?p=2296 The big software vendors claim they can sell you an "out of the box" data warehouse including reporting that installs onto your Oracle, Microsoft or SAP ERP package. Are these things really that easy? Is it actually cheaper to build it yourself? How do you determine if a given product is a good fit for your situation?

Here's the secret- If you answer yes to all of the following its probably a good idea to buy:

• You've configured your Enterprise Resource Planning (ERP) software exactly as the vendor who wrote the data warehouse or data mart expected you to.
• You use ALL the modules of the ERP- and do not have any "best of breed" applications for any areas.
• You are interested in the key performance indicators that they have included and they calculate them the same way your company does
• Your end users like the layout, formating and color choices in the reports included and have promised not to ask for changes.

In fact if all this is true, then you will probably save HUGE amounts of money by using a pre-built package.

But. Lots of things tend to get in the way of that dream. More often then not it's not the pre-built package that needs to be focused on- it's what you have in your system and how you've configured it.

ERP Customization


Anyone who has been involved in a large ERP project knows that there is always at least a bit of customization. If the ERP is a good fit to your industry, and the project is focused on avoiding customization and using all those "flex fields" and "category codes" as they were intended, then the pre-built data warehouse that is available for that ERP might be a very cost effective route.

But here's the rub; in many ERP projects rather than change the existing processes (which by the way is often the justification for installing an ERP- process improvement), the business users find reasons to "keep things as they are"- and as a result, the ERP bends to the existing process rather than the other way round.

This means that new tables are created to hold information that normally would be in a different format in one of the standard ERP tables. This means that that code in field "X" that normally would mean something is overloaded to mean three or four things. This means that field "Y" that isn't needed by your company ends up being used for something completely different.

Well, when the pre-built data mart plugs into your ERP, it doesn't know anything about those tables you've added, and its assuming that the data is in the standard tables. Its already configured its ETLs, and star schemas to use those two fields that you've high-jacked, and so all those reports and cubes don't make any sense without modifications.

To be fair, many of the pre-built data warehouse frameworks have evolved to have lots of flexibility and configurability in them- you can modify and map the ETL jobs provided to fit your data. You can turn off KPIs so that they don't load in false data.

But the trick is the more you have to modify, map and configure, the more effort you spend. The more you shut off, the less you have. In the end you can find that you are always trying to fit your specific requirements into the general, vanilla requirements that came out of the box. Often, this configuration is more effort and demands more compromises than just building what you need.

Non ERP transactional systems

In some ERP implementations, certain modules of the ERP are not used and instead a third party software provides the functionality. Often, key information is then moved in and out of the ERP as required for transactions via an enterprise application integration (EAI) tool. In many cases, the reason the additional system is used is because it has more functionality (and therefore stores more detailed and varied information) than the ERP module. As a result, the EAI tool cannot move all of the information into the ERP, only summaries and selected data. This means that for the functional area that is treated by that module at least some of the detail users are interested in does not exist in the ERP. And if it's not in the ERP, it won't be picked up by the pre-built solution.

Diverse and specific report requirements

Its amazing how the layout (column order, summarization, which key performance indicators where) of a report can be very very important to report consumers. Its also amazing how much work can be required to modify the hundreds of reports and cubes that came with the pre-built system. Report writers everywhere are nodding their heads. Enough said.

Data Quality Issues

Even if you've used all the tables and fields in the ERP exactly as needed, and the reports look just fine to your users, if they have not already been given access to all this information you will probably find that there is lots of data quality work to be done. This cost is there for both the build and the buy options- just be aware of it, because it means the buy option is going to be more than just the sticker price.

One ray of hope- very specific industry solutions

Hold on, the vendors say, but we have industry specific data warehouses and data marts- these are built just for your industry so they know your business. We have one for banking data marts, another one for pharmaceutical data marts, you can get a sales data warehouse specifically for retail, and so on.

Without a doubt the more focused the pre-built data warehouse or data mart is the more likely it is that its going to save you time and money.

Conclusion? Often when something sounds too good to be true...

The bottom line is, although you can tell I'm not head over heals for prebuilt datawarehousing, I think the field has evolved. But then again, so have the prices.

The key is to be realistic regarding how much modification will be required now, and going forward, and how many of those hundreds of cubes and reports that are included are actually going to be applicable to your business.

It is possible that a package exists that will provide the best value. Just be very very careful when you are comparing the costs- because its never quite as sweet a deal as it first seems.

Smaller, very focused pre-built data marts, particularly targeted to applications or modules that are used fully by an enterprise are probably much lower risk and more likely to provide real value than a massive "out of the box, install it in weeks not years" data warehouse.

My personal experience with pre-built products was that in the end, when the dust settled, we had touched almost every ETL job, and the reports that were the most used and useful were ones we had built from scratch, largely using data loaded out of custom ERP tables.

If you have customization in your ERP, you have to think hard about this. The reason the custom tables were put in there was because they hold information that was important enough to justify the development. This probably means that the information in them is going to be important to the business intelligence you are building.

]]> http://www.datamartist.com/build-vs-buy-the-pros-and-cons-of-premade-data-warehouses-and-data-marts/feed 0