But you don`t allow them to come see your property, and your sketch doesn't specify anything about the existing landscaping- just the final configuration. Do you think the landscapers would be willing to offer a reasonable price ?

Unlikely. What if there are existing patio stones to remove- or an in-ground swimming pool that`s got to go?

No landscaper would take on a job without understanding the lay of the land, and the existing conditions. It would be impossible to estimate the job. Anyone who did would give you a huge price to cover themselves, or demand extras upon discovering the extra work.

Yet when companies hire consultants to build them business intelligence solutions, or do data migration, it often happens with only the roughest outline of the existing data sets. Certainly, often a data model is included- but knowing what the table SHOULD contain rather than what it does is just not the same thing. It never ceases to amaze me that the simple, cost effective practice of data profiling is just often not part of the initial phases of so many business intelligence and data migration projects.

With the right data profiling tool, and just a few days work, its possible to gain a huge amount of insight into the data quality in your systems, and as a result, be able to make radically more accurate estimates of the cost to go from the "as is" to the "to be".

Phil Simon talked about this in a great post on the Data flux blog called "What Consultants Don't tell you", and raises an important and somewhat ugly truth- many times, service providers don't WANT to do data profiling because it reveals the true extent of the work to be done, increasing the budget requirement, and makes the project less likely to be approved.

Now certainly, we can't use a broad brush to paint all consultants, but it does lead to a reduction in the number of times valuable tools such as data profiling are recommended even though in my opinion they are a low cost, no-brainer, do it unless you are crazy first step to any major project.

You are going to spend potentially millions of dollars on a business intelligence or data migration project- spend a few weeks to look at the data with the right tools first for goodness sake!

If you want to get a reasonable cost estimate, and you want to go into your business intelligence or data migration project with open eyes, don't imagine you can know what it will cost to get from here to there if you don't take a good look at where here really is.

Full disclosure- of course, you are reading the Datamartist blog, and Datamartist has lots of data profiling functionality- so you have to understand that we are incredibly biased on this topic. If you are able to overlook our inherent bias, give the tool a try- you`ll discover things about your data you might not have wanted to know, but its better to face the truth prepared, than to rely on wishful thinking, and then discover the bad news when you're well into the project, and your budget is almost gone.

The majority of the development cost in the current business intelligence methodology is often in getting the data out of source systems (Extract), and transforming it to make it consistent across all the various dimensions needed (Transform) and then putting it in a model that is easy to query and analyse (Load). The creation of these ETL jobs is made dramatically harder if the data in the source systems is not consistent.

Change is the challenge

Companies are not static- they grow, diversify, change strategies, reorganize, rename and restructure. They acquire other companies or are acquired. The structure and content of the data their systems often tells you this story, and if the proper work is not done to keep the data consistent with itself and the new situation then this story will be painful and complex.

Remember ten years ago when we acquired company X, but decided not to change their customer codes to our standard, so all the codes had an "X" prefixed so that we wouldn't have duplicates? Well, those X's are still there, and all our queries have to deal with multiple code structures.

Remember how we used to have three independent databases, one for each region, then when we went to the new data center and put everything into a single database, we ended up with multiple schemas and all those crazy views rather than consolidating into a single instance?

When the data migration project made the decision to reduce the project cost by not addressing data consistency, they simply pushed this cost in the future, most likely turning a one time expense into an ongoing and expanding annual business intelligence cost.

You end up with crazy ETL jobs that parse the same field in different ways depending on the date of the transaction, or on other fields- "If the transaction is before 2002, then the first digit of the product code means X, otherwise it means Y, unless of course its from the western division, who do it differently so then you need to look at field A and use the CASE statement..."

Reduce Business Intelligence cost through data cleanup

If your data is cleaner you'll reduce business intelligence cost across your entire BI architecture.

• Reduce ETL and report development cost- both initial, and the cost of ongoing maintenance. Every change request will take more time if all the models are complex due to underlying data complexity.
• Reduce hardware costs- complex queries require more processing, and bigger servers to meet that nightly load window
• Reduce time spent reconciling numbers. Complex ETL means that chances are business intelligence reports don't match up easily with the operational reports from the source systems. People will spend time constantly double checking these discrepancies, and it will undermine confidence in all data.

Fix the problem at the source. Not in the Business Intelligence.

Business intelligence is far too often left to fix all the issues in the source systems- and then becomes the focus of dissatisfaction when costs and delays become unacceptable.

I've heard people argue "Thats what ETL is for right? Why are you complaining?"

Assuming that the ETL will fix the sins of the source system is an inefficient and costly strategy.

Everything is a balance, perfection does not exist, but when deciding what to fix and what to leave, don't let a lazy data migration project saddle you with years of business intelligence costs- when it's time to bulk load data into the system, make it as right as you can.

The transactional systems that a company uses can be the glue that binds, or can be a key part of the walls that block inter-departmental collaboration and information sharing.

The data migration project is your best chance to break down the data silos- but it won't be the easy path.

Not only can the ERP project create new opportunities for efficiency and process improvement, but it can also be the process through which people from across the company start to work together.

ERP projects often require teams of subject matter experts from various functional areas (finance, sales, manufacturing) to work together- depending on how serious your silos are it might be the first time many of these people have met.
Make the most of it- see if you can't encourage some new working relationships that last beyond the project. A few key personal contacts between departments can make a huge difference.

Best case and worst case: big difference

1. Best case- processes that stumbled along without any coordination are totally reworked for the better and everyone- including your customers- see a huge positive difference that ends up hitting your bottom line. Win!
2. Worse case- no-one will accept change, everyones position is that either things stay the same, or others adapt to their vision of the future- you end up finding a way to shoehorn all the existing processes into the ERP, resulting in a messy, customized compromise that no one likes and limits progress while costing a fortune. So sad.

If you are doing a data migration project- how can you help make it be a more positive and unifying step, rather than a transfer of the existing silos intact into the new ERP at great expense?

The first step is to admit you have a problem.

The first thing to do is to identify how bad your silos are, and get all the players to look the problem in the face.
If you have silos, but people don't identify it as an issue, there is no way you'll get the resources you need to fix them.

Make it clear that customization is the enemy.

Sure you could customize the ERP application that you are installing to do exactly what each department does now. Add some tables, write some bolt on code.

I'm sure you can do it. Thats not the point.

Technical resources love customization because it's more fun than configuration. Don't fall into the trap of thinking "we're satisfying the customers requirements."

Customization in an ERP system is expensive, usually reduces functionality and in the long term drives significant maintenance and upgrade costs.

If you customize the new ERP system to accommodate the existing silos you are NOT satisfying requirements. You are leading the business towards a failure that they don't really understand.

You need to find language that the functional teams and management can understand and explain the risks clearly.

If we don't get together and consolidate our processes and our data definitions we're going to end up with a mess in the ERP.

The system was not designed to do that- the point of an ERP is to be integrated- if our departments don't work together, than the ERP isn't going to make anything better- and its a waste of money.

Get top down support for the painful change that is necessary.

Sometimes things can be grassroots, originating at a level below the executive suite. If that works for your company, thats great.

But in the majority of cases the kind of short term pain that breaking down entrenched silos will cause is too intense to be initiated anywhere but at the very top.

The leadership needs to make it clear that everyone is going to share in the change, and no-one has a "get out of change free" card.

It has to be clear that it is not a competition between the various approaches that might exist, but a move to a new approach.

It's really really hard. If it's not hard, you're not doing it right.

There's nothing I can write here to make this kind of change easy, it's not.

But by building cross functional teams, making goals clear, and managing expectations it's possible to make progress.

So in summary what I've been trying to convey in this series of posts:

• Data migration projects are data quality projects.
• Data migration projects are master data management projects.
• Data migration projects are an opportunity to break down the walls between silos within the organisation.

You can make a real difference if you strive to make the data migration project advance on these three fronts.

You always have to work within the constraints you have in terms of budget, company culture, existing systems and of course, internal politics, but if you take a step by step and pragmatic approach with these goals in mind, you'll be contributing to the solution.

]]> http://www.datamartist.com/data-migration-part-5-breaking-down-the-information-silos/feed 0 http://www.datamartist.com/data-migration-creating-a-data-dictionary-how-to-tackle-master-data-management http://www.datamartist.com/data-migration-creating-a-data-dictionary-how-to-tackle-master-data-management#comments Thu, 17 Dec 2009 20:52:39 +0000 James Standen http://www.datamartist.com/?p=3696 Migrating data is complicated. It's particularly hard because of course it's not just a physical move. Data definitions are different from the legacy to the new systems. To get this right you need to manage these data definitions.

In this post, I'm going to discuss some things to keep in mind during this process. As with the other posts in this series, I'm not going to be talking about specific tools or getting into super technical discussions.
I'm going to assume that you do not have millions of dollars to spend on the state of the art master data management software and its configuration. Instead, I'm going to present the high level concepts, and focus on some of the change management aspects. How can you focus on what's important, and avoid having master data derail your data migration project?

This post is number four in a series. If you are a linear type, and want to read all the posts in order, part 1 is here, then two, and three.

There is a reoccurring theme here- just as when I discussed in part two about how a data migration project is also a data quality project because often existing data quality issues must be resolved;

A data migration project is also a master data management project.

Since often the legacy systems were department based, the concept of formal data definitions, and the processes needed to manage them across functional boundaries simply don't exist. But if your data migration project is moving master data from multiple legacy systems into a single new ERP application, you are going to need at least a basic set of processes to manage it going forward.

Welcome to change management, cross functional teams and data governance committees.

Data migration means integrating data that used to live apart, and is owned by different groups within the company.

Often, a data migration project is part of a new ERP project to combine a number of legacy systems (Finance, Sales, Manufacturing) into a single integrated application. Because all of the legacy systems were independent, often the data definitions used are very different, and any mappings or reconciliations exist only at a high level if at all.

So what does this mean?

• There will be technical challenges:
  • Data will be in strange and wonderful (or at least many) databases from various vendors.
  • Data will have different codes and structures
  • Data will be stored at different granularity
  • Data will be stored in different units
  • You may have things like time zones, date formats etc. to deal with
• There will also be actual definition type challenges
  • Although something is refered to as "X" in more than one system, the definition may be very different.
  • Different functional groups look at data in fundamentally different ways- engineers vs accountants, sales people vs human resources professionals.
  • People will be attached professionally, emotionally and politically to their definition and view of things.

The technical challenges are significant- but its the definition challenges that are often the most difficult, because they involve people. These often represent fundamental changes to the processes within a company, and require coordination across many departments, and the political empires that often exist. It takes a firm hand, and good executive support.

Where to start? Well, know some of the potential pitfalls, and define realistic goals clearly.

Just because we call something the same thing doesn't mean its the same thing.

I've seen people go into a meeting, have a one hour discussion regarding definitions, come out with a high level list of the metrics and a real belief that they agree on everything. They tell you "Your data migration is going to be straightforward, we're pretty much on the same page."

Don't be fooled by this.

After 30 minutes of digging into the details in the two different systems, you will realize that there are lots of things they don't agree on.

Why did they think that they agreed? Because they spoke the same words, but meant completely different things. I say, a car has four wheels- you nod. We're done. But was it a sports car, a sedan, an SUV? I say car- I see my car, you say car you see your car. We nod. Meeting over.

Doesn't work- you have to dig into the details, and put them on the table.

Details are important, and everyone hates them. Find the detail people and get them together.

No one likes the details, but the heart of any master data management work is in those nuts and bolts. Start to dive into the details and most eyes will glaze over. Find the people who care because the details affect their daily job.

What you need to do is make a "data definitions" working group made up of people from multiple departments.

1. Get at least one detail oriented team member from each department
2. Make sure they are respected and experienced- you need people who know their stuff.
3. Make sure it's clear to all departments that the output of this working group will be the data dictionary used by the new system.
4. Add in one or two data people- someone who can query all the systems involved, and can both answer the detailed questions from the group, but also actually validate concepts as they are created by making prototype extractors and transformations
5. If you are missing key expertise, bring it in. Even if its your core business, if you think you're not following industry best practices for your definitions, nomenclature or processes, don't just "keep doing it the way we do it".
6. take advantage of industry standards for naming and coding, this can sometimes also be a method for avoiding internal battles- we're going to follow the standard, not have a homemade coding system.

So now you have to give this working group a goal.

The solution is not always to have only one definition in the end.

What? Isn't that the whole point?

Yes it is when there really is only one definition, but don't chase after "a single version of the truth" ignoring the reality of what people need, and what metrics are used in the business.

Just because finance and manufacturing calculate "X" in two different ways does not necessarily mean that you have to stop using one of the definitions. Both groups might have a perfectly valid reason to calculate it they way they do. The key is- don't use just one name for it. Its NOT "X".

There are two metrics "X1" and "X2" and they have different definitions for different reasons. Put them both in the dictionary, make it clear what the differences are, and when each measure is used.

It's not a battle. Everyone is on the same team here.

People of all sorts often tend to see things in terms of winning and losing.

You might find people thinking in terms such as "Finances definition for "X" has to win. We know how it is supposed to be calculated, and we are going to set it right." Avoid framing the debate as a competition- present it from the start in an inclusive way;

"We need to list all the metrics and definitions involved, as used by all departments- and if there are duplicates we'll consolidate, but overall we want to capture them, and ensure they are available. We can all acknowledge that each department has different needs owing to their functional area."

Create a central repository for the dictionary- make it public from the start.

Now, if you have the million dollar master data management (MDM) softward solution chances are it has this functionality and more. If you don't, then you should plan to make an internal web site or equivalent means of providing access.

I won't go into the detail of what an entry in this data dictionary actually consists of, partly because this post is already past my usual length limit, but mostly because there is no hard and fast rule. You can go anywhere from a basic dictionary (that describes what each metric is in english) all the way to a highly sophisticated meta data management tool that interacts directly with your extract transform and load (ETL) logic actually changing how numbers are transformed in your migration jobs. (I'm not sure I'd go that far, even with that mythical million dollar tool.)

The bottom line is, be pragmatic- and be public.

In my experience it has been critical to create a published, work in progress dictionary from the start of the process. The ideal is to have an online database that allows contributors to edit and comment directly. This is important for a number of reasons:

1. It makes the process transparent- everyone can see what the definitions are going to be.
2. It makes the scope and complexity of the challenge visible. By having all the definitions on line, everyone can see just how many definitions there are, and how much detail there is.
3. It tracks versioning and ensures that all the work is captured. Don't have people with lots of excel files on their local hard drive holding up your progress.
4. It lets you publicly assign responsibility for each defintion. DO NOT have a librarian who is reponsible for all definitions - you want to have a process owner from each department assigned to each definition. The ideal number is hard to find and will depend on the company, too few and you don't have buy in, to many and it's not managable.
5. It allows metrics to be viewed by everyone. Which department has defined and approved the most definitions? Who is holding up the effort? A bit of inter-departmental competitiveness might be useful in this regard.

Define the processes needed to maintain and expand your data dictionary into the future

Finally, as you capture the definitions you need to perform the data migration, the ideal is to define and kick off the processes and required organisational groups that will continue to manage the dictionary after the project go live.

In master data management circles this is often called data stewardship and its critical that you have at least a basic formal process. In the past, when a given chunk of data was only being used by one department, changing how it was defined would be understood by the department, and could be managed somewhat informally. Now, when the same bit of data might be used accross the company since everyone is in the same system, changing that query or how the value is calculated at the request of one department could create all sorts of issues for others. There must be a process in place to validate the change, and to communicate it so that everyone in the system can be aware when the numbers shift beneath them.

In summary, it's really really hard.

The bottom line is, master data management requires a pragmatic, step by step approach, and is never finished. Manage everyones expectations, and be very careful as to what you promise your data migration project can achieve in this area.

Very large big bang type projects are probably not a good idea, even if you are armed with the latest and greatest tools. As Wikipedia states in its criticism section:

The value and current approaches to MDM have come under criticism due to some parties claiming large costs and low return on investment from major MDM solution providers.

Much like massive data warehouse projects, risks are high, particularly if you don't have near fanatical support from the very top.

The best approach is most likely to build step by step, clearly defining what has to be defined for the data migration project at hand, and putting in place realistic processes that can be augmented and enhanced over time as your company begins to "grok" why having a data dictionary is important.

It's a long road, but ignoring master data management in your data migration project adds risks to your projects success, and makes it even harder for future master data efforts to succeed.

Just as a data migration project is an opportunity to affect data quality for the better, it can also be a positive influence on your master data.

Next post- I'm going to wrap up this series with some discussion about how important it is to forge a collaboration between the information technology (IT) department and the various departments within a business, and how IT can play an important role in building bridges through the silos- both data silos and others

]]> http://www.datamartist.com/data-migration-creating-a-data-dictionary-how-to-tackle-master-data-management/feed 3 http://www.datamartist.com/data-migration-part-3-mapping-the-legacy-systems-meta-data-and-application-mapping http://www.datamartist.com/data-migration-part-3-mapping-the-legacy-systems-meta-data-and-application-mapping#comments Mon, 14 Dec 2009 18:17:07 +0000 James Standen http://www.datamartist.com/?p=3640 This is part three of an ongoing series that's taking a look at data migration projects. In this part we're going to talk about how important it is to know where you are starting from, before you head off on a new application journey. Understanding and mapping your legacy systems is a key success factor for a data migration project, but can be a very difficult and time consuming battle. In this post, I'll talk a bit about some approaches I've found useful in my experience.

If you like, you can start with Part one which was a light hearted introduction to data migration projects in general, and part two, where we talked about the importance of data quality.

Why are we spending so much time on this? Thats the OLD system- we need to focus on the future!

Here are just some of the important things the legacy mapping needs to clarify:

1. Data location- You can't migrate data if if you don't know what it is and where it is.
2. Data dependencies to other systems All processes and interfaces that rely on interfaces to the legacy systems need to be either replaced or shut off. Often this means that even if the new system is not involved, other systems may stop working because they get data from the legacy systems. The data migration project is not just about turning on the new system. The consequences of turning off the old system have to be known and managed.
3. Legal requirements to keep legacy data available. Even if data is not migrated to the new system there may be additional data migration requirements into data warehouses or documents that have nothing to do with the new application.
4. Infrastructure dependencies. The actual infrastructure that the legacy systems are on might perform other tasks that although not directly related to the legacy system will cause issues when that infrastructure is removed. (For example, someone installed a service of some sort on one of the servers that is used by other applications that are completely unlrelated from a data point of view).

Often the first time the Legacy system is documented is just before it's shut down.

Despite our best intentions, sometimes documentation doesn't get updated. This is the reality for many systems, and particularly for legacy systems.

One of the first steps in a data migration project is to gather all the existing documentation for the legacy systems, and all the systems they talk to, and make sure its accessible to the data migration project team.

It is critical to have tight control over these documents, and to ensure that everyone works off a "live" version- because your mapping is going to update that documentation, and every developer, data modeler and application team member needs to know that they have the best and latest version.

The application interface diagram.

Now, the ideal situation is to have a dynamic, self correcting, scanning Configuration Management Database tool (CMDB tool) that already has every scrap of meta data about every application and all its interfaces ready to go.

If you have one of these, good for you, and you can stop reading.

For the rest of us, lets talk practical methods of mapping what we have.

How to get the data.

1. Scan the environment- catch the interfaces in the act.
   • Monitor network traffic to detect exchanges between applications.
   • Scan file systems to find interface files and determine frequency.
   • Catalog services and activity of those services on servers.
2. Get out there and talk to people.
   • Ask people- where is data from this system used?
   • Look at management reports and trace backwards to find where information is pulled.
   • Don't assume the interface is direct. My record discovered is 6 hops from source to the excel sheet used by the CEO, with the information passing through two of the same systems twice.
   • Hunt down people that were involved in the original installation. Often they'll have key information that can save you time.
3. Any other way that works.

What to do with it.

If you don't have a complex tool to do the mapping of all your systems, then one approach that is a step above the "lots of excel sheets and powerpoint slides" approach, is to use a tool like Microsoft Viso. I've used it successfully to map applications, by having the drawing and the interfaces BE the database. This ensures that everything in the drawing is on the interface list, and everything on the interface list shows up on the drawing.

1. Create different objects in Viso, and give them attributes. At a minimum you need an application, interface and database object.
2. Draw the applications and the interfaces between them in a single large viso drawing, and fill in the attributes in the visio objects.
3. Make some simple VBA code in the drawing to dump all the data into flat files or excel sheets (or directly to a DB if you get ambitious).

It's simple, but it is far better than having spreadsheets, and a drawing- and then constantly trying to determine if the two agree with each other.

In the ERP project where I used this technique, we identified over 1500 interfaces between hundreds of application instances. The ERP project was a very large effort with hundreds of project resources, and multiple phased projects implementing a new common system. The actual original mapping took two people about 3 months to do. They had to work with about 30 different applications support people to systematically map all the applications, and the interfaces, one by one.

A key part of the job was to actually validate the documentation. IE if the documentation said there was a chron job that ran a script on server X, actually go to server X and watch it run. This meant that we could be confident in the map, and make plans based on it.

Everyone on the team used the drawing and lists generated from the drawing to stay on the same page. And it was a big page- the key is to also have access to a plotter- we were plotting out a pretty good size wall poster by the time we were done.

The ERP teams had the drawing taped up to the wall- and they were making notes right on it and emailing my team. We would update the master, and publish a new version, along with the generated lists.

In building this drawing, we found that most of the interfaces were "under" or "un"-documented, and that if documentation did exist, generally it was wrong. By establishing the "official" document for the legacy systems, we focused and coordinated the design effort in a way that would not have happened, if each team just had their own marked up copy of the original documentation or the part that was of interest to them.

Having the map means you can make the plan

This drawing and the interfaces mapped were critical in planning the migration.

1. Create different layers in your drawing for each phase "Phase 1", "Phase 2", "Phase 3", or "Feb 2010", "Aug 2010", "Jan 2011" etc.
2. Hide or show systems and interfaces (including the new applications and interfaces) as they were phased in or out for each layer.
3. By viewing and printing layers separately, you can see a step by step plan for the migration- with your application architecture and integration map at each phase.

This was a powerful tool to both do the planning, and to make sure everyone understood the timing and sequence. With multiple phases over a three year period, the project needed it, and without such an overall view, such critical planning would have been haphazard.

The challenge with this mapping is to find the right level of detail required. Not detailed enough and it is wasted effort. Too detailed and it will consume excessive resources and time.

A simple approach- What talks to what and what it runs on.

There are two key aspects to mapping your application architecture.

1. Functional relationships- applications talking to other applications, with interfaces between them.
2. Infrastructure relationships - which servers, network connections, services and databases are involved in the functional relationship

You can't show both completely on a single drawing- don't try. Some applications run on multiple servers, many servers run more than one application, data bases are shared by many, interfaces often use common infrastructure such as EAI tools etc.

The approach we took, and it worked well, was to show the functional relationships on the diagram, and hold the physical relationships (which databases were on which servers/clusters and which application ran on which server etc.) in the attributes of the applications.

We did sometimes show some physical attributes on the diagram for easy reading, but only as an annotation- the relationship was done via the attributes in the visio application objects.

This meant that you could ask "What runs on this server?" and could ask "Which servers are involved with this application?" by doing a filter or query on the data. Very useful things if you are planning to shut down a server. You make a checklist, and one by one make sure everything is either shutdown, or moved.

Here's a simple example to illustrate what the diagram might look like;


The circles with the numbers were the interfaces, each one had attributes like "To" , "From" and "Method" etc. The level of detail you go to is a function of how ambitious you are, but at a minimum you need to record the fact that the interface exists.

So in summary:

1. Create a single map of all your applications and interfaces and share it with everyone on the team
2. Make sure you validate your map carefully, looking into the actual systems, and talking with as many people as needed to ensure you have captured everything
3. Make a step by step plan for the migration, showing when each application, interface and infrastructure item is phased in or out.

Next up- the data dictionary and how do we get everyone to agree on those definitions?

]]> http://www.datamartist.com/data-migration-part-3-mapping-the-legacy-systems-meta-data-and-application-mapping/feed 0 http://www.datamartist.com/data-migration-part-2-determining-data-quality-is-the-first-key-step http://www.datamartist.com/data-migration-part-2-determining-data-quality-is-the-first-key-step#comments Mon, 07 Dec 2009 15:54:34 +0000 James Standen http://www.datamartist.com/?p=3517 Any discussion on data migration needs to include data quality as a core topic. Migrating data from one set of applications to another, particularly when the applications were never designed to interact, and share little or no common structure or definitions is a complex task. This task is made even more complex by the data quality issues that almost always exist in legacy systems.

In part 1, we set the stage for data migration with a lighthearted look at the pitfalls of badly migrated data. In this post, I'm going to talk about data quality, and give some perspective on why its so important to be thinking about data quality every part of the way during a data migration project.

It's not just a data migration project. It's a data quality project feeding into a data migration project.

Because the destination system will often have a "stricter" data model, or may use a broader data set due to enhanced features it is likely that bad quality data in the legacy system that is "not causing any problems" will become an issue in the new system.

Lets look at a trivial example. In the existing customer relation management system, there is a mandatory field for the customers birth date. Sales people are supposed to use this to help maintain a personal relationship. Its mandatory, but doesn't get used by the system, except for a reminder getting sent to the sales person on the customers birthday.

So sales people have to enter in the date, it gets marked as missing data if they don't.

So what happens? They fill in some random date, and ignore all the reminders.

But in the new system, the customer relation management system sends an email with a discount code to the customer on their birthday, wishing them a happy one and offering them a discount. The birthday in the system is now directly linked to business transactions with financial impact.

Birthday is now a field that suddenly needs to be addressed.

This is perhaps a silly example, but there can be more serious ones- the bottom line is that often the data migration project will be required to actually improve data quality at the same time. Its two projects in one- make sure you've budgeted for it, or you're going to have to somehow pull them both off for the price of one. Not easy.

It seems obvious, yet remarkably, data migration project budgets seem to be created based on the assumption of near perfect data.

So the first step in any data migration project- in fact ideally before the project budget even gets fixed:

Do a first pass data quality audit as early as possible.

Know what you have. Profile the data in the legacy systems so that when it comes time to migrate the data, you know where the problem spots are in the key fields.

What do I mean by first pass? Well, this is early, so you may not have access to a mature design for the target system (the "to be" application architecture and configuration) so that means you may not be able to focus on the specific data that the new system will be requiring, and in what format, or what granularity it is required.

Never the less, you can be pretty sure that things like order history, customer information, product attributes etc. will be there, so your quality audit of the legacy system is going to pay off.

As more detailed specifications are created, and a better understanding of both the source and target systems evolves, you will need to continue to assess the impact of data quality on your migration efforts.

But you may find that even just doing a data quality audit of the legacy system raises a whole new set of questions. The adventure often starts with a bunch of blank looks when you ask for the legacy systems data dictionary.

So next up- Establishing the meta data model. Because in migration one of the first steps in getting somewhere is knowing where you are starting from.

This post is part 2 of a series- part 1 is here. and part 3 is here.

]]> http://www.datamartist.com/data-migration-part-2-determining-data-quality-is-the-first-key-step/feed 0 http://www.datamartist.com/data-migration-part-1-introduction-to-the-data-migration-delema http://www.datamartist.com/data-migration-part-1-introduction-to-the-data-migration-delema#comments Tue, 01 Dec 2009 15:12:37 +0000 James Standen http://www.datamartist.com/?p=3477 The world is a dynamic place. Businesses change. Companies merge, technologies shift and applications come and go.

Our data, however, is often less dynamic. All those sales records in the old sales system don't morph into the format required in the shiny new ERP. The data has to be dragged kicking and screaming into the new format.

Thus, the art and science of data migration is one of the most challenging, and most important, of the information technology black arts. In the next few posts, I'm going to take a light hearted, but hopefully useful look at data migration.

So imagine you are in a company that, for whatever important, unavoidable, and "it was super clear at the time" reason is going to change its sales system. The new system is just so much better- as the VP of marketing points out:

"The new sales system will drive synergies and encompass our core strategy for value creation and customer focus."

Oh good.

What are the possible strategies in regards to moving your data?

Abandon your data

Sure, just shut down the old server, send the hard drive to the recycler, turn on the new system empty, and wait by the phone for the next order.

"Hello, Acme lots of products, can I help you?"
"You'd like to place an order, ok, can I get your company name and address please."
"Yes, I know you've got a customer number, but we don't use those any more, they're from the old system. We have a new one now."
"You want product 23432- that's an old code, let me try to find the new one, just a sec..."

Probably not going to work out so well.

Enter all your data by hand over the weekend.

Depending on the amount of data, you might actually be tempted to do this. Sit a bunch of people in front of a bunch of computers, give each of them a stack of reports from the old system, and show them how to enter the data into the new system. Buy them pizza. Look really stressed as 6am Monday morning approaches.

The "advantages" of this approach:

• Because the new system thinks that you aquired every one of your existing customers over a two day period your companys growth rate looks fantastic.
• You get written up in computer science journals as a case study on the error rate for manually keyed data. The researchers are particularly excited about the clearly defined ramp affect built in by the progressive sleep deprivation of the data entry clerks.
• You get lots of practice calculating customer abandonment metrics as over the next six months data errors cause undelivered invoices, mis-routed orders and incorrect product configurations. You get excellent data on what makes customers most angry, and drives them to switch to your competitors most quickly.
• Even though the new sales system doesn't work out so well, the reduction in transactions makes it possible to save lots of money by reducing the number of servers and the amount of storage needed in the data center.

Or, you could launch a data migration project

Yep. Probably what you should do. But how?

Seriously, what are some things to consider in a data migration project?

Over the next few posts, I'm going to continue on the theme of a sales system migration, and talk about strategies that can help get you through your project, and the kinds of "Gotchas" that can spring up. Every data migration project is different, but hopefully I'll shed some light on the process, and share some of the experience I've had.

For me, the number one thing to think about in a data migration project is the impact on your customers.

Remember those customer people? The ones that give us money and expect something good in return? Turns out they're the cornerstone of your business.

Minimize the impact on your customers

Now, I'm not saying that you shouldn't consider other stakeholders too, and there is always political and financial constraints to any large project, but if it comes down to a choice between making life difficult for someone inside your company, vs a customer- well, someone has to take it for the team, and the customer isn't the right choice.

A badly executed migration project can cause enough issues for customers to push those that were wavering to the competition. A really badly executed one can push loyal customers away.

Depending on how cut throat your industry is, it could cost you significant marketshare, maybe even end up being the start of the end.

On the other hand, a well executed migration to a system that not only cuts your costs but provides real value to the customer could be a key part of your competitive advantage.

Next post: A data migration project is never just a data migration project- it's a data quality project too.

]]> http://www.datamartist.com/data-migration-part-1-introduction-to-the-data-migration-delema/feed 0