Sometimes, we make decisions NOT to set a goal, because we don't want to break it.

You might be thinking you really should step up your data quality monitoring- get some data profiling underway to help identify the data domains and areas you most want to tackle in 2012. But you might be also thinking that with all the pressures and cutbacks that many companies are facing, you don't have the resources to implement a full scale profiling and monitoring effort, and so might decide to delay.

Don't wait. Just do it. The perfect is the enemy of the good.

Rather than worrying about how much of your data you are going to be able to cover, or that you can't devote enough resources to tackle all of your reference areas at once, work at the problem from another direction.

First, start with master data.

Master data is the data that all your other data is made from. It's the data everyone uses to view the massive piles of transactional data, so one bad row in a master data table, and the impact is felt across perhaps hundreds of reports, and multiple time periods. If you have a product in the wrong category, then every transaction, across perhaps hundreds of customers, and all time, will be mis-catagorized, and every total, sub-total and calculated metric using it will suffer.

While bad transactions are bad, bad reference data is deadly. Bad reference data takes a good transaction and messes it up.

Worst first!

Make a list of your reference tables/area. Customer, Product, Chart of account, etc. etc. What are the most important for your business? This isn't something I can tell you- you have to think about what is most critical.

If you are a company that purchases large amounts of materials from many vendors, and purchasing decisions are fast paced and critical, then maybe it's your vendor master, and your accounts payable.

On the other hand, if you have lots of interaction with your customers, and errors in the customer master cost you business, then start with that.

The key is to first make the list, and then think to yourself "if I have bad quality data, where am I most afraid it will be?" Start profiling there. You want to find the worst first, and fixing that will have the greatest positive impact.

Get to know your data

Don't worry about setting complex or work intensive goals right away. Data profiling is about data discovery sometimes. You need to wade into your reference data, play with it, tease out patterns and relationships. As you get to know your data, you will be able to better identify where there are issues to tackle, and where root causes might lie for data quality issues.

One approach might be to simply resolve to spend an hour a week, every week, profiling some data. If you aren't do that now, you will find that even just a bit of time set aside will give huge insight- sometimes we get too busy to do the basics, and we miss opportunities to make significant improvements with relatively little effort in our data.

You know that all that data means something- but what?

One of my favorite code-busting stories is the epic victory over the Enigma code during the second world war. One of the reasons its of interest is that it was one of the early applications of computing- but the key lesson I think is from not the brute force computation done, but the strategies used to crack the code.

When you are trying to crack a code, one of the key things you need are "Cribs"- some way have samples of coded message and clear text. These cribs can radically reduce the number of possible ways a code can be decoded.

In the case of enigma, the allies would listen for German U-boat radio transmissions, while also using direction finding equipment to estimate their location. Standard procedure was for a U-Boat to first radio a weather report.

By painstakingly back tracking known weather conditions and locations of U-Boats when they transmitted it was possible to take advantage of that first weather report- there were only so many ways to say "Sunny and calm". Having this crib gave them a way to break into the code.

What is the point in terms of Data profiling? While it's critical to have the right tools to analyse the data (a data profiler like Datamartist, for example), its also important to get out there and talk to people, understand whats going on- collect some Cribs that will help it all make sense.

Note: In a previous post, I introduced regular expressions and went through a simple example using Canadian Postal code validation.

In this post, we'll learn some more regular expression syntax, and explore some examples.

Using regular expressions to validate codes

Often, within ERP systems, various entities such as organisational units, products, etc. are assigned structured alphanumeric codes. These codes have a defined structure- all valid codes must have this structure. For example, a product code might have the pattern:

aaa-nnnnn

Where the first three characters are a product group code, a mandatory dash, and then a five digit product number.

You will remember from last time, a set of acceptable characters is defined using square brackets. So digits only can be specified by [0123456789] and letters by [abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ].

To simplify things, dashes can be used to include all digits or letters in a range, and these to formats can of course be combined. The string [a-zA-Z0-9] specifies that any letter or digit is acceptable.

In addition, you can make a rule as to how many consecutive characters are required that follow the rule by adding curly brackets. So the following pattern would test for a valid product code in this format:

^[a-zA-Z]{3}[-][0-9]{5}$

The {3} and {5} apply a length limit to the immediately preceeding test- so there must be exactly 3 letters, a dash and then five numbers.

To specify a range of lengths, you can put two numbers in the curly brackets, separated by a comma. So to allow product codes like ABC-1, DFG-12 or HGF-34564 we can use {1,5} instead for the second test- this allows the number after the dash to be made up of between one and five digits.

^[a-zA-Z]{3}[-][0-9]{1,5}$

From this example, you can see how its possible to fairly easily make patterns to test for a wide range of structured codes.

Defining more than one rule set at a time

Sometimes, you might have two different formats possible for a given value. As painful as it is, companies often change their coding rules, but due to legacy constraints, keep legacy coded entities in their data sets as well.

In regular expressions, it is easy to combine two completely different patterns in one test by using the pipe character ("|"). In fact, this operator can also be used within a single pattern to check for one OR the other of two different things.

For example, lets pretend that we've added a whole new product line to our offering, and the codes for these new products will have the structure XY-NNNNN-AA and XY is either the letter "A" followed by a single non-zero digit (ie A1,A2,A3...) OR if X is any letter other than A then Y should be a letter too. (I'm just making this up, but I can tell you some of the rules I've seen in the real world are a lot more complex and seemly arbitrary than this.)

A regular expression that would validate this new product code would be as follows:

^([aA][1-9]|[b-zB-Z]{2})[-][0-9]{5}[-][a-zA-Z]{2}$

Note that we put the first expressions all in some curved brackets, and separated two different rules by the "|" to make an or. So if [aA][1-0] OR [b-zB-Z]{2} is true at the start of the string, then that part of the test is OK. By starting at "b" we exclude the bad codes of "AB" or "AG" etc. because the rule says if it starts with A, the second character needs to be a number.

But, of course all of our old product codes are still going to be there, and they won't pass this new test- so to combine the two, we just take each of them, and put a pipe between them:

^[a-zA-Z]{3}[-][0-9]{1,5}$|^([aA][1-9]|[b-zB-Z]{2})[-][0-9]{5}[-][a-zA-Z]{2}$

It looks like crazy gibberish- but when you build it a bit at a time, it all makes sense. And now, you can easily detect if there are any product codes that don't conform to either valid structure.

Using this code in Datamartist

If you were using the Datamartist tool, you could find the bad product codes by adding a new column using a calculation block, called, say "ProductCodeValid", that was defined as the following Datamartist expression:

REGEX([PRODUCT],"^[a-zA-Z]{3}[-][0-9]{1,5}$|^([aA][1-9]|[b-zB-Z]{2})[-][0-9]{5}[-][a-zA-Z]{2}$")

This column will now have TRUE for the records where the product code is well formed, and FALSE where there are issues.

Regular expressions are a very useful way to check many types of data quality and they can help you avoid all sorts of crazy tests with LEFT, RIGHT and MID string functions.

Using REGEX will let you create much more powerful data quality code.

If you want to try building some REGEX yourself, you can download the Datamartist free trial, and give it a go with your own data.

They way regular expressions work is like this:

1. A pattern is defined. This is a string of symbols that act as a set of rules.
2. A text string to test, and the pattern are given to a regular expression engine, and compared.
3. The engine returns a true/false value meaning the string follows the rules or not ("PASSED" or "REJECTED by the pattern)

This is obviously very useful for someone interested in Data Quality- If you had a pattern that said "Is this a valid email address?", and got PASS or REJECT back, it would give you a good idea as to the quality of that field in your contact database.

One advantage of regular expressions is that because they are widely used, lots and lots of them have been created, detecting all sorts of patterns- meaning that while you can write your own, you can also look up useful ones you need in libraries.

Regular expressions aren't magic, of course- the result is only as good as the program. (As always.) Depending on how well the regex is written (or not) there may be false positives or negatives.

A regex example- Canadian Postal Code

Lets look at a simple example, and see how they work. Being from Canada, I'm going to use the example of validating a Canadian postal code.

Canadian Postal codes take the format ANA NAN, where "A" is a letter and "N" is a number. So what we want is a regular expression that will return TRUE for valid postal codes, and FALSE for postal codes that just can't be right. "K9J 2K2" could be a valid postal code, but we know that "38X AB2" just can't be.

In a regular expression, we use anchors to say where to start matching. In this case, we want just the Canadian postal code, so we'll use the anchor character "^" to specify the beginning of the string.

To specify that a character has to be within a given set or range of characters, we use square brackets. So to match when ever the first character of a string is a letter, the regex would be:

^[a-zA-Z]

This regex will return TRUE for all strings that start with a letter. Thats fine, but not yet specific enough for a Canadian postal code (we Canadians are very very picky).

So we can add a number, then another letter constraint to our pattern.

^[a-zA-Z][0-9][a-zA-Z]

So far, now any string that starts with ANA will result in true- we're almost half way there! Next, we want to specify that the space is optional- that is, its acceptable to have the space or not.
To do this, we use the "?" to specify that the space is optional. And then to finish up, we add the part of the expression that detects the NAN, and end with a dollar sign which specifies that that needs to be the end of the string (otherwise all strings that started with a valid postal code would pass);

^[a-zA-Z][0-9][a-zA-Z][ ]?[0-9][a-zA-Z][0-9]$

So there it is- or is it?

But is the REGEX fussy enough?

While this pattern that we've created does detect the ANA NAN pattern, and even allows the space to be optional, if you know Canadian postal codes ,you'll know that in fact ANA NAN is not enough by itself. There are only certain letters that actually exist in certain locations. So a better REGEX pattern for Canadian postal code validity would be the following:

^[abceghjklmnprstvxyABCEGHJKLMNPRSTVXY][0-9][abceghjklmnprstvwxyzABCEGHJKLMNPRSTVWXYZ][ ]?[0-9][[abceghjklmnprstvwxyzABCEGHJKLMNPRSTVWXYZ][0-9]$

This pattern explicitly lists valid letters. Canadian postal codes do not use the letters D,F,O,Q or U anywhere and they do not use W or Z in the first position. Of course, this brings up another issue with any data quality method- remember Canada post could decide to change the rules- then your data quality test would need to be updated.

Ok, so that means the Postal code is ok right?... uh, No.

So this regular expression will detect if a text string of length 6 or 7 is a valid Canadian postal code- but remember that this alone is probably not enough. Chances are that this postal code is stored as part of an address, which will also include the city and province. In Canada, postal codes are of course unique to a given province- the first letter defines the area, and each area exists within a particular province (large provinces have more than one letter assigned to them).

This means that a properly formed postal code could be invalid- for example, an address in Quebec that has a postal code that starts with the letter "V" which is for British Columbia has clearly got something amiss.

So while learning a bit about regular expressions, we've also learned that probably if you had a big mailing list to clean you would probably want to use a dedicated tool- postal addresses are an area of data quality where lots of attention has been paid over the years, and writing a lot of custom logic and regex patterns is probably not a good use of your time. But for application specific codes and strings it might be very useful. In my next post, we'll look at some more tricks with regular expressions that can be used to analyze data quality.

I've posted a small collection of useful regular expressions to the datamartist website here.

Datamartist V1.3.0 PRO and Regular expressions

The professional edition of the Datamartist tool provides a function REGEX(text,regex expression) that returns TRUE or FALSE depending on if the text "matches" with the regular expression specified. This function can be used anywhere in Datamartist where expressions are available, making it a powerful way to test if a string matches one or more patterns.

Rather than looking at the actual individual values for a given column, by profiling the structure of the values you can understand at a higher level the quality of data. This technique is primarily used for string based data.

Can't find the forest because of the trees?

Lets look at a simple but very real example. You have a data set with a few tens of thousands of rows that includes a text field for telephone number.

What kind of phone numbers do you have?

Nobody wants to look through 40,000 rows value by value to see where the bad records are.

Profiling with character substitution and elimination

With a data profiler you can define rules:

1. IF the character is a digit (0-9) then replace it with the letter "n"
2. IF the character is a letter (a-z) then replace it with the letter "a"



These two simple rules turn 40,000 rows of phone numbers into a short list of different phone number formats- and we can see which ones seem valid or seem to have problems.

By adding one more rule, eliminating a few characters that do not directly affect the phone number, we can reduce the list even further.


The last rule ignores the space, open and close brackets and dash characters- and that simplifies the different formats in the analysis to a mere five formats;

1. 10 digits
2. 11 digits
3. 9 digits
4. A 7 character string (drilling down by clicking on this bar reveals the word "unknown"- not so useful.)
5. Missing.



Of course, data format profiling does not always give a final answer regarding data quality. In this case, just because a phone number has the right number of digits, does not mean that it is a valid phone number, and even if its a valid phone number it may not be the right phone number for that customer...

Data format analysis for structured codes

This same technique is also useful for any fields that are meant to contain structured codes.

For example, say a valid product code is supposed to start with one letter, where the first letter should be one of A,B,D or G be followed by a dash, and then four digits.

So A-2324 is a valid code, but M-2334 and J234 are not, for example.

The following rules would help detect a validly formatted string (and point out the issues:)



The result on a mocked up set of data:



Using data format patterns to examine the contents of a string column is a very useful way to start to understand what's in the column. More than just a Yes-No result, it actually gives you a visual look at what types of issues exist.

The analysis here was done using the Datamartist tool, an easy to use data profiler and data transformation tool. To try making some data format rules for your own data, give the free trial a try.

Up next in our data profiling series of blog posts- an even more powerful (although often more complex) technique called "Regular Expressions" or "Regex". This specification language can define complex rules that analyze strings and determine if they belong to a given set (say, "Valid product codes" as in this example) or not.

]]> http://www.datamartist.com/data-profiling-rules-and-data-formats/feed 0 http://www.datamartist.com/data-profiling-avoidance-a-cautionary-tale http://www.datamartist.com/data-profiling-avoidance-a-cautionary-tale#comments Wed, 22 Sep 2010 17:02:13 +0000 James Standen http://www.datamartist.com/?p=5133 In which Bill decides that data profiling is not necessary.

We sit down to keyboard to tell a sad tale
Of data project manager Bill, and his epic huge fail

Bill thought he had all that it took
He looked through the data model, "It reads like a book"

He happily flipped through tables and maps
Overjoyed to see a model so free of all gaps.

Then through the sales system checking the norm
Oh Joy! Oh Bliss! validation for each form!

"Why, golly" he said, "I'm really in luck"
"With all of this checking the data won't suck!"

He knew data quality was a number one killer,
But with a model like this and all those good rules,
The data would be clean, perfect, tables of jewels.

He told his boss- "Hey, piece of cake!"
His boss was most happy at how little time it would take.

Then late one night, with a knock on his door,
In came his development lead, eyes to the floor.

"Something is wrong, and I don't understand it."
"The product code extractor, well, it's not like we planned it!"

The codes were all wrong! But how could that be?
Bill went out to the data entry folks to take a look see.

"Oh those codes?" They said, not a care in the world.

"We just enter "X" then "4" and then double bee."
"The checker always accepts that you see"

"We can't spend time looking up codes!"
"We have orders to fill, we have to ship loads!"

"Besides the info you want is in a spreadsheet we use."
"When they ask us which system, it's that one that we choose."

Bill's boss was most thoughtful when told the bad news
He thought about all the money and time that he'd lose

"Bill" he said, a frown on his face.
"Your assumptions are a total disgrace"

"You need to look at what data is there."
"When it comes to the model, the real world does not care."

While getting HIS boss on the line with his auto-dialer,
Bill's boss said to poor Bill "Don't just stand there! Go get a profiler!"

Don't Ignore your data.

Don't you be like Bill! Check out the Datamartist data profiler.

Look your data right in the face.



]]> http://www.datamartist.com/data-profiling-avoidance-a-cautionary-tale/feed 1 http://www.datamartist.com/when-should-you-data-profile-morning-noon-and-night http://www.datamartist.com/when-should-you-data-profile-morning-noon-and-night#comments Wed, 22 Sep 2010 13:16:26 +0000 James Standen http://www.datamartist.com/?p=4955 Data profiling is an important part of any data related project. The question often arises when the best time to data profile is. As you would expect from a software company that sells a really cool visual data profiling tool, our view is "all the time".

Using data profiling tools before the project

Data profiling is useful even before the project is defined. By doing a first higher level data profiling on key data sets, you will get;

• better project scope definition
• a more accurate budget estimate
• a clear baseline from which improvement can be measured
• the ability to correctly manage expectations

The last two are important ones. Making wild promises based on what the data model says should be in the tables and then failing to deliver due to data quality issues is not nearly as career enhancing as defining an ambitious but doable scope and delivering based on the actual data, clearly communicating the progress made based on facts.

Data quality issues can seriously (double digit percentage seriously) affect the final cost of a project. Knowing about issues will let you set a realistic budget.

Data profiling at the beginning of the project

After the initial higher level data profiling done before the project, budgeting for a more detailed data profiling of the source data will:

• allow clear design guidance to ETL developers
• Clearly identify the subject matter experts needed to understand the data, and let you engage them early- rather than in a rush when ETL development hits the underlying data issues, and the project is already running late and over budget.

Data profiling during the project

• By setting up automated data profiling tasks for the output of each ETL process, ETL developers and architects can track the progress for migration, cleansing or conversion tasks using concrete information.
• Objective criteria can be set for each profile task to determine what level if data quality is considered "acceptable" for the final data load or fact set deliverables.

Data profiling at the end of the project

• Doing a final data profiling run, and comparing it to the baseline established before the project will provide a clear Before/After view that will both clearly communicate the progress made, but also assist in justifying and promoting the next data quality initiative.
]]> http://www.datamartist.com/when-should-you-data-profile-morning-noon-and-night/feed 1 http://www.datamartist.com/what-is-data-profiling-and-when-and-why-should-it-be-done http://www.datamartist.com/what-is-data-profiling-and-when-and-why-should-it-be-done#comments Tue, 21 Sep 2010 20:44:36 +0000 James Standen http://www.datamartist.com/?p=4491 First of all, full disclosure, if you haven't already noticed, this blog is written by a software company that makes a pretty cool data profiling tool. We've just released our new version V1.3.0, so obviously we can't pretend to be completely objective in the "should you data profile debate". But bear with me, because I think I'm still going to be able to convince you it's a good idea, regardless of which tool you select.

The way I think of data profiling is that it is the "Reality check" of all your data activities.

The data model is the theory.

Data profiling lets you see reality.

The bottom line is that you know what is supposed to be in the column by looking at the data model, but you can know what is actually in the column by data profiling the data.

For a fantastic detailed look at data profiling, check out this series of blog posts from Jim Harris, over at obsessive compulsive data quality. It's a great series that clearly communicates the type of analysis that might be done while doing data profiling and the kind of detective work it sometimes takes. While I'll be looking at a number of the same concepts in this series of blog posts, being a software vendor, I won't be as tool agnostic as Jim was- we'll be demonstrating the techniques using the Datamartist tool.

You can download a free trial of the Datamartist tool and try the profiling techniques and features yourself as you follow along.

We hope you enjoy the upcoming data profiling tutorials and that you give the Datamartist data profiler and transformer serious consideration for your data profiling needs.

]]> http://www.datamartist.com/what-is-data-profiling-and-when-and-why-should-it-be-done/feed 1 http://www.datamartist.com/datamartist-v1-3-0-value-distribution-data-profiling http://www.datamartist.com/datamartist-v1-3-0-value-distribution-data-profiling#comments Mon, 26 Jul 2010 18:33:50 +0000 James Standen http://www.datamartist.com/?p=4855 This video gives a quick (under two minute) look at the Datamartist data profiler's ability to explore the distribution of numeric values in a data set by counting the number of values that fall into a series of equal size buckets. It highlights the datamartists calculation, visualization, selection and drill down features using a simple example.

This value profiling tool is just one of many of the Datamartist data profiling tools capabilities, download the free trial of the BETA to try all the functionality with your own data.

]]> http://www.datamartist.com/datamartist-v1-3-0-value-distribution-data-profiling/feed 0 http://www.datamartist.com/data-profiling-do-it-do-it-now http://www.datamartist.com/data-profiling-do-it-do-it-now#comments Fri, 07 May 2010 02:11:58 +0000 James Standen http://www.datamartist.com/?p=4496 Imagine that you have bought a new home, and you've decided to do some landscaping. So you pick three landscapers, draw a rough sketch of what you want, and ask them to bid on the job.

But you don`t allow them to come see your property, and your sketch doesn't specify anything about the existing landscaping- just the final configuration. Do you think the landscapers would be willing to offer a reasonable price ?

Unlikely. What if there are existing patio stones to remove- or an in-ground swimming pool that`s got to go?

No landscaper would take on a job without understanding the lay of the land, and the existing conditions. It would be impossible to estimate the job. Anyone who did would give you a huge price to cover themselves, or demand extras upon discovering the extra work.

Yet when companies hire consultants to build them business intelligence solutions, or do data migration, it often happens with only the roughest outline of the existing data sets. Certainly, often a data model is included- but knowing what the table SHOULD contain rather than what it does is just not the same thing. It never ceases to amaze me that the simple, cost effective practice of data profiling is just often not part of the initial phases of so many business intelligence and data migration projects.

With the right data profiling tool, and just a few days work, its possible to gain a huge amount of insight into the data quality in your systems, and as a result, be able to make radically more accurate estimates of the cost to go from the "as is" to the "to be".

Phil Simon talked about this in a great post on the Data flux blog called "What Consultants Don't tell you", and raises an important and somewhat ugly truth- many times, service providers don't WANT to do data profiling because it reveals the true extent of the work to be done, increasing the budget requirement, and makes the project less likely to be approved.

Now certainly, we can't use a broad brush to paint all consultants, but it does lead to a reduction in the number of times valuable tools such as data profiling are recommended even though in my opinion they are a low cost, no-brainer, do it unless you are crazy first step to any major project.

You are going to spend potentially millions of dollars on a business intelligence or data migration project- spend a few weeks to look at the data with the right tools first for goodness sake!

If you want to get a reasonable cost estimate, and you want to go into your business intelligence or data migration project with open eyes, don't imagine you can know what it will cost to get from here to there if you don't take a good look at where here really is.

Full disclosure- of course, you are reading the Datamartist blog, and Datamartist has lots of data profiling functionality- so you have to understand that we are incredibly biased on this topic. If you are able to overlook our inherent bias, give the tool a try- you`ll discover things about your data you might not have wanted to know, but its better to face the truth prepared, than to rely on wishful thinking, and then discover the bad news when you're well into the project, and your budget is almost gone.

]]> http://www.datamartist.com/data-profiling-do-it-do-it-now/feed 2