The transactional systems that a company uses can be the glue that binds, or can be a key part of the walls that block inter-departmental collaboration and information sharing.

The data migration project is your best chance to break down the data silos- but it won't be the easy path.

Not only can the ERP project create new opportunities for efficiency and process improvement, but it can also be the process through which people from across the company start to work together.

ERP projects often require teams of subject matter experts from various functional areas (finance, sales, manufacturing) to work together- depending on how serious your silos are it might be the first time many of these people have met.
Make the most of it- see if you can't encourage some new working relationships that last beyond the project. A few key personal contacts between departments can make a huge difference.

Best case and worst case: big difference

1. Best case- processes that stumbled along without any coordination are totally reworked for the better and everyone- including your customers- see a huge positive difference that ends up hitting your bottom line. Win!
2. Worse case- no-one will accept change, everyones position is that either things stay the same, or others adapt to their vision of the future- you end up finding a way to shoehorn all the existing processes into the ERP, resulting in a messy, customized compromise that no one likes and limits progress while costing a fortune. So sad.

If you are doing a data migration project- how can you help make it be a more positive and unifying step, rather than a transfer of the existing silos intact into the new ERP at great expense?

The first step is to admit you have a problem.

The first thing to do is to identify how bad your silos are, and get all the players to look the problem in the face.
If you have silos, but people don't identify it as an issue, there is no way you'll get the resources you need to fix them.

Make it clear that customization is the enemy.

Sure you could customize the ERP application that you are installing to do exactly what each department does now. Add some tables, write some bolt on code.

I'm sure you can do it. Thats not the point.

Technical resources love customization because it's more fun than configuration. Don't fall into the trap of thinking "we're satisfying the customers requirements."

Customization in an ERP system is expensive, usually reduces functionality and in the long term drives significant maintenance and upgrade costs.

If you customize the new ERP system to accommodate the existing silos you are NOT satisfying requirements. You are leading the business towards a failure that they don't really understand.

You need to find language that the functional teams and management can understand and explain the risks clearly.

If we don't get together and consolidate our processes and our data definitions we're going to end up with a mess in the ERP.

The system was not designed to do that- the point of an ERP is to be integrated- if our departments don't work together, than the ERP isn't going to make anything better- and its a waste of money.

Get top down support for the painful change that is necessary.

Sometimes things can be grassroots, originating at a level below the executive suite. If that works for your company, thats great.

But in the majority of cases the kind of short term pain that breaking down entrenched silos will cause is too intense to be initiated anywhere but at the very top.

The leadership needs to make it clear that everyone is going to share in the change, and no-one has a "get out of change free" card.

It has to be clear that it is not a competition between the various approaches that might exist, but a move to a new approach.

It's really really hard. If it's not hard, you're not doing it right.

There's nothing I can write here to make this kind of change easy, it's not.

But by building cross functional teams, making goals clear, and managing expectations it's possible to make progress.

So in summary what I've been trying to convey in this series of posts:

• Data migration projects are data quality projects.
• Data migration projects are master data management projects.
• Data migration projects are an opportunity to break down the walls between silos within the organisation.

You can make a real difference if you strive to make the data migration project advance on these three fronts.

You always have to work within the constraints you have in terms of budget, company culture, existing systems and of course, internal politics, but if you take a step by step and pragmatic approach with these goals in mind, you'll be contributing to the solution.

]]> http://www.datamartist.com/data-migration-part-5-breaking-down-the-information-silos/feed 0 http://www.datamartist.com/datagov-looking-at-the-us-governments-data http://www.datamartist.com/datagov-looking-at-the-us-governments-data#comments Tue, 26 May 2009 02:42:48 +0000 James Standen http://www.datamartist.com/?p=2279 The Obama administration has taken another step in making government data available online in launching the Data.gov website. Whats on the site right now, and what can we do with it?

The website has two catalogs of information available- the "raw data catalog", and the "tools" catalog. At the time of posting this, there are 47 entries in the raw data section, and 27 in the tools section. For this post, I focused on the "raw data catalog".

Thinks I liked:

• The site lets users rate data sets- giving an Overall rating, and then rating on Data utility, Usefulness and Ease of Access (giving you a vote of 0 to 5 stars). Democratic. Thats keeping with core values.
• Each data set came with some extra info, including data dictionarys- for the example set it can be found here.
• The fact that the data WAS raw. Better to get it out there, then do something to it that makes it less useful.
• Although Catalog doesn't yet have a lot of entries to need it yet, there is keyword search and filters by categories, government agency and file type.
• The US Federal government isn't afraid to use the word "Potty" when it needs too.

POTTYAGE,Age of the most used toilet ,1,27,Numeric
POTTYLEAK,Does the most used toilet leak,1,28,Numeric
POTTYPARTS,Any of the part of the most used toilet replaced,1,29,Numeric
POTTYFIX,When was the most used toilet repaired,1,30,Numeric

Things I found disappointing:

• The data dictionary information was often in PDF format, when the contents were in fact a dimension table. Having this data in the form of a data file would save us some parsing. Here was the example pdf for the featured data set, for example.
• Related data sets were not grouped together- the catalog only had one data set per entry, even if some were directly related or joinable. It would be much more powerful to have the data sets grouped together. (This would also let us group together those dimension like tables if they were available.)
• Just not enough data yet, so I found the mish-mash of various data sets a bit bizarre. I didn't have any pressing need to analyze Community collaborative Rain, Hail and Snow observations, or the Migratory Bird Flyways, or the detail of the world wide earthquakes in the past 7 days (although admittedly that last one is kind of cool).

Bottom line is any data is better than no data, and if there's something there you're interested in analyzing, then good on the US government to make it available for you. There is not a lot there now, but the hope is that over time more federal CIOs will feel the pressure to get their data available.

Whats going to be key, however, to make this all work is to establish a standard for both data and meta data that is more robust than throwing some csv files and PDF data dictionary documents on a web site. I look forward to seeing how this evolves.

Update: Sunlight labs, with sponsors including Google have a contest for the best analytical applications based on data.gov data.

Doing a deeper dive into the data with the Datamartist tool

I couldn't resist taking a peak using Datamartist.

The featured data set that was displayed prominently on the Data.gov home page was the "Residential Energy Consumption Survey (RECS)" for 2005. The description of the data file is as follows:

The Residential Energy Consumption Survey (RECS), which is conducted every four years, provides national statistical survey data on the use of energy in residential housing units including physical housing unit types, appliances utilized, demographics, fuels, and other energy use information. This dataset (i.e., the full RECS dataset) is very large in size and may require specialized software to open on your computer. The file might not open completely in Excel 2003 or earlier versions.

The file ends up being very WIDE rather than tall- over 1000 columns, and 4382 rows, totaling about 10 Mb in a csv file.

I did one pass to make a sub set file focusing in on a few columns, then I built some mini dimension tables to join up by cutting and pasting out of the PDF. Once I had the data in datamartist I connected up a join block and checked it out in the data profiler:



Oddly, when investigating POTTYAGE, I found that the majority of the rows had the value zero, which was not listed in the data dictionary. Hmmm. Perhaps even the feds have data quality problems now and again.

]]> http://www.datamartist.com/datagov-looking-at-the-us-governments-data/feed 0