Sometimes, we make decisions NOT to set a goal, because we don't want to break it.

You might be thinking you really should step up your data quality monitoring- get some data profiling underway to help identify the data domains and areas you most want to tackle in 2012. But you might be also thinking that with all the pressures and cutbacks that many companies are facing, you don't have the resources to implement a full scale profiling and monitoring effort, and so might decide to delay.

Don't wait. Just do it. The perfect is the enemy of the good.

Rather than worrying about how much of your data you are going to be able to cover, or that you can't devote enough resources to tackle all of your reference areas at once, work at the problem from another direction.

First, start with master data.

Master data is the data that all your other data is made from. It's the data everyone uses to view the massive piles of transactional data, so one bad row in a master data table, and the impact is felt across perhaps hundreds of reports, and multiple time periods. If you have a product in the wrong category, then every transaction, across perhaps hundreds of customers, and all time, will be mis-catagorized, and every total, sub-total and calculated metric using it will suffer.

While bad transactions are bad, bad reference data is deadly. Bad reference data takes a good transaction and messes it up.

Worst first!

Make a list of your reference tables/area. Customer, Product, Chart of account, etc. etc. What are the most important for your business? This isn't something I can tell you- you have to think about what is most critical.

If you are a company that purchases large amounts of materials from many vendors, and purchasing decisions are fast paced and critical, then maybe it's your vendor master, and your accounts payable.

On the other hand, if you have lots of interaction with your customers, and errors in the customer master cost you business, then start with that.

The key is to first make the list, and then think to yourself "if I have bad quality data, where am I most afraid it will be?" Start profiling there. You want to find the worst first, and fixing that will have the greatest positive impact.

Get to know your data

Don't worry about setting complex or work intensive goals right away. Data profiling is about data discovery sometimes. You need to wade into your reference data, play with it, tease out patterns and relationships. As you get to know your data, you will be able to better identify where there are issues to tackle, and where root causes might lie for data quality issues.

One approach might be to simply resolve to spend an hour a week, every week, profiling some data. If you aren't do that now, you will find that even just a bit of time set aside will give huge insight- sometimes we get too busy to do the basics, and we miss opportunities to make significant improvements with relatively little effort in our data.

This version of Datamartist brings with it some useful new functionality, including new functions that can be used in expressions, new capabilities in terms of exporting to databases, and a new data block.

In this post, we'll look at two new features, the Pivot block, and the enhanced database export capabilities.

Pivot Block

Our beta testers loved this new block. The pivot block lets you do the equivalent of a cross-tab query, rolling up a measure, and distributing the value in a new set of columns, where the column names are provided by the input data set.

Here is a simple example showing how it works:

Say we start with a set of data that has mutiple rows for each date, and different values in the color field, and a quantity measure:

Then we can connect one of the new pivot blocks to this data set like so:

The pivot block lets us select which columns to include (this defines the level of detail to roll up to), which string column to use to generate the new column names, and which measure to use as well as the rollup method (sum, average, min, max)

The result? The output of the pivot block looks like this: now we have a summary by color for each date, with a column for each color value.

Database export enhancements

Now, when exporting to a database, there are a number of new options.

One of the most interesting is the capability to execute SQL commands in the database either before and/or after the data is exported into the table.

This provides the capability of running stored procedures, or launching follow on database side processing after Datamartist writes the data into the DB.

This is a powerful new capability, and makes it even easier to integrate datamartist into various systems, and get your data quality and profiling data where you need it.

If you haven't checked out datamartist yet, we're not sure what you are waiting for- download the free trial, and give it a go.

Data quality super powers.

Fighting the anti-data forces of evil.

Data silo fun.

The joys of a moving target

Data migration tools.

While it's a complex subject, some basic rules of thumb can avoid the deeper rabbit holes.

The first thing to understand about Data Quality rules is they aren't as easy as they may look. Data is in theory something in the ordered world of computers, but in reality is in the "flexible" world of humans. A huge amount of data is entered by members of the group "Homo sapiens" (or mutilated by software written by members of that group) and as a result is not as ordered as we would all like.

The challenge for data quality practitioners is to remove the chaos injected by those highly involved primates (us) and make the data the sterile, ordered, never any question about anything type that we all imagine in our fantasies.

But how?

In the end, it is amazing how powerful and complex the various solutions to this problem are.

But I suggest that there are some basic principles that can help guide us.

First- do no harm.

One of the risks of any data quality initiative is that it actually screws up the data more. Don't define rules that are so complex, and so sure of themselves that they actually make the data worse. Be humble. Don't change data unless you are pretty sure it's a good idea. Err on the side of not screwing up the original. And keep a copy of the original- so if things do go off the rails you can undo- or at least try to understand what when wrong.

Go out and talk to the people

Don't sit in your ivory tower and speculate as to what the data means. Go out there and watch people enter it in. See what real world type things are happening that never make it into bits and bytes.

Attack the basics first

Focus your first efforts on dealing with the basics- they will resolve the vast majority of the issues- don't chase after the outliers until you have the "easy" cases taken care of- the tough stuff is a case of diminishing returns- look first at how to fix processes and train your people to make the majority of typical data entry cases more accurate before you start looking into artificial intelligence based hyper-multi-semantic-algorithmic-learning-matching-holistic-flux-capacitor data quality systems.

Less is more- the fewer rules the better.

So whats the rule about making rules? Try to make less rules, and test them in a pragmatic way. It is possible to have so many rules that the rules themselves have data quality issues- don't go there.

Sometimes the simplest things will bring the greatest benefit.

In the coming weeks, I'll be posting about how to design, implement and monitor Data quality rules using the Datamartist tool.

We've had lots of great feedback from our customers and are thrilled with how people are using Datamartist, not just for powerful and flexible data profiling, but for data migration, data quality work and ad-hoc datamart creation.

As always we're committed to continually improve our products- Here are just a few of the features added in this latest version;

Block definition import and export

Datamartist adds a new level of reuse-ability and collaboration capabilities with the addition of block export/import.

This lets you export a block configuration (or a number of blocks, with all their connectors) to a file that can then be imported into any other Canvas, either by yourself or by your colleagues that are also using Datamartist. Just select the blocks you want, and right click to export- just right click anywhere on any canvas to import an existing block file.

We've found this particularly useful in saving filters, segmentations, and even full data profiling blocks to give us a library of useful blocks and block groups that we use again and again.

Improved database connectivity and connection management

We've also added some features and improved how we connect to databases in Datamartist

Datamartist can connect to SQL Server, Oracle, MySQL, MS Access, Text files and Excel files, as well as having and ODBC driver that lets you connect to many other databases. Now its even easier to manage a large number of database connections for multiple database types- keep all those servers and connections at your fingertips when you are combining all that data!

Data profiling in an affordable, graphical, visual environment

Find out why people are loving the combination of an ETL and a Data profiling tool in one- using Datamartist not just for data profiling, but for data migration, data quality audits, and ad hoc datamart creation. Try the free trial today.

Having technical things like tools work is useful, and good. But just because you build it, does not mean they will come.

The challenge often in Data Quality is that often what has to change even more than the technology or tools is the behaviours and perspectives of the people in the organisation with data quality issues. At the very least, the users have to use the tools. Very few data quality solutions are of the "full autopilot" bad-data-goes-in-here-good-comes-out-here type.

As much as we engineers would like to solve everything with software, people are involved in Data Quality.

While a fantastic bit of data profiling analysis or an elegant and powerful data transform would seem to be enough, the truth is sometimes how and when you present these things is key to getting the non-engineer people to buy in.

Sometimes preparing people over time, and introducing things in a step by step way helps them understand, and makes the technology and the change required less daunting.

Because I'm looking out my window at a tentative (very tentative it's only March after all) spring day here in Toronto, I'm going to use a summer barbecue analogy.

The tools and technology are the steak. The steak is key to the party. In the end (at least for me in this analogy) the steak delivers most of the value in your summer BBQ party value proposition, but you'll have more guests and be more successful over all if you package the whole.

Sometimes, part of selling the steak is the sizzle, the preparation, the things around the steak.

It's the smell of the BBQ getting ready, it's the sound of the steak hitting the grill- its the cold drink, the conversation, the games on the lawn for the kids.

In the end, even if you know that 90% of the deal was that steak, if you just put a steak on a plate and give it to each guest the moment they arrive, its just not going to get the same response.

In my usual round about way the point I'm trying to get to is that you can't solve technical problems, then drop them on people desks and say "do it". You need to invite them to the party. Prepare them for the menu, ask preferences, give them some time to hear the sizzle, smell the charcoal, enjoy the sunshine in expectation of that steak.

Steak is good. Remember to plan some sizzle too.

Obviously, we have to believe something. But today I'm asking you to be skeptical. Question your facts.

Let me give you an example. I'm a Canadian, and looking out my window right now, I can see a pretty healthy snow fall accumulating. Lots of the white stuff. Brings to mind the fact that some cultures in the far north have over 100 words for snow.

Hang on. Is that a fact? Tell me- have you heard a variation of that?

I'm sure I read that somewhere. I've heard others mention it. It makes sense- I mean, people living in the far north would see lots of snow, and would know all about it, and so their language would evolve to encompass lots of different qualities of snow.

Sounds good.

Only, is it? It's an idea that "just makes sense". People seem to just accept it as soon as you say it. People are likely to pass the idea along to others- because it makes a compelling story.

But in fact, it's wrong. I'll let you google to your hearts content if you like to find more evidence than my say so, but after reading a number of articles on the subject, (here is an example, and of course the Wikipedia entry.) it seems clear that there are not 100 words for snow in any language. In fact, English has about the same number of ways of talking about snow as languages from societies in the far north.

So the point of all this is- what myths do you have in your organisation? Things that "everyone" knows are true. Things that when they are explained to you make "perfect sense". Things that you teach to every new hire so that they "know how things are".

The insidous thing about "facts" is that once they gain purchase, any contrary evidence tends to be called an "exception", or discounted.

Use data to find out what is true. Fight to improve the quality of your data to find more and more new truths. Question the status quo if the data contradicts it. Don't assume that something is wrong with the data when "things don't make sense." Maybe they don't make sense because your assumptions are just plain WRONG.

Be very aware that you might be making decisions based on myths that while sounding so plausible, so clear, so common sense, are pure fantasy.

The good news is that all your competitors might be doing the same thing. If you look at your data, and see through it, you might show them all how wrong they are.

The trick with public data like this is that often (ok almost ALWAYS) either data is missing, or the codes don't match up. In this case, the country descriptions from various data sets I want to use don't match- and different data sets have different holes (i.e. not all datasets include data for all countries). Finally, some data sets have a different definition of a country- for example, they break out places like "British Indian Ocean Territories" that need to get rolled up in the UK numbers.

Country statistics data such as GDP, GNI and Population were also incorporated to provide dimensions to carry the analysis out on. The raw trade data was obtained from Industry Canada's "Trade Data Online" (http://www.ic.gc.ca/sc_mrkti/tdst/tdo/tdo.php?lang=30&productType=HS6). The World Bank was the source of the country statistics data (http://data.worldbank.org/indicator). A zip file containing the raw data, as well as the Datamartist data transformation .dmc file is provided at the bottom of this post.

I started by transforming the country statistics data. The raw data included information on GDP, GNI, Total Population and Urban Population. A screenshot of the Datamartist canvas for the first portion of this data transformation is provided below.As seen above, the first step in the data transformation involved importing the four excel data files. During this import, columns with data not relevant to the year 2009 were filtered out and zeros were inserted into any null data rows, signifying that data for that row was not available. A series of data "Join" functions were then carried out to create one data file containing all of the country statistics information. Upon completion of joining these data files," a "Calculation" block was utilized to replace any null data values resulting from the data join with zero's. Finally, the country statistics information was joined with a country cross reference list. Basically, this join standardizes all of the country names.

The second part of the country statistics data transformation focused on segmenting the data, as shown in the Datamartist canvas screenshot below.Before the data could be segmented, it was summarized so that there was only one data row for each standardized country name. A "Calculation" block was then added to calculate the GDP per Capita, GNI per Capita and Urban Population Percentage using the Population data. With these calculations complete, a series of "Segment" blocks were added to the canvas. The "Segment" blocks are extremely useful because they add an additional column to the data set which is populated according to a set of segmentation rules defined by the user. In this example, the "Segment" block was used to segment the GDP per Capita, GNI per Capita, Urban Population Percentage and Population data. The segmentation rules for the Population "Segment" block are shown in the screenshot on the left.

A similar set of data transformations was also carried out on the Canadian Import and Export Trade data. As seen in the Datamartist canvas screenshot, the Canadian Import and Export Trade data was imported, joined and null data values were replaced with zeros. The country names were then standardized and summarized so that the Canadian Import and Export Trade data could be joined with the Country Statistics data.

With all of the raw data transformed into a suitable format, a final set of data transformations were carried out to create a single text file. This text file was then exported so that QlikView could be used to analyze the data. A screenshot of the Datamartist canvas for this final set of data transformations is shown below. In this final set of data transformations, the "Star Schema" block was used first. The "Star Schema" block is a handy data transformation tool because it allows numerous data join operations to be carried out simultaneously. It was used to combine the Country Statistics data and Canadian Import and Export Trade data with data defining a country's geographical region. A screenshot of the "Star Schema" block configuration window is shown to the left. The joined data was then put through a "Calculation" block one last time to eliminate any null data values. Finally, the transformed data was exported as a text file so that it could be analyzed in QlikView.

The transformed data was then imported into QlikView and a dashboard was created to analyze the data with. QlikView is a great data analysis tool because it allows data to be filtered and visualized very efficiently. In this example, I made a dashboard that allows Canada Import and Export Trade data to be visualized using the Country Statistics Data segments created using the Datamartist software as filters. A screenshot of the dashboard with no filters applied is shown to the right. I am now going to show a series of screenshots with different data filters applied. To start off, I want to see the countries that Canada Exports the most goods to. To do this, I just dragged a box over the largest bars on the export graph as seen below. Once I finished making the data selection box on the graph, I released the mouse and QlikView automatically zoomed in on the area I selected. In addition, the upper right table in the dashboard updates as well. The image below shows the results. As seen in the image, Canada's biggest export trade partners in 2009 were the United States, the United Kingdom and China. As a second example, I will filter the data using the data segments created in Datamartist. If I click on "Asia" in the "Region" box then only data from the countries in Asia is shown in the table and graphs. Furthermore, the segments in the other filter boxes (GDP per Capita, GNI per Capita, etc) updates to the region selection as well. It does this by highlighting the data filter segments that are valid for the "Asia" region in white. For example, in the "GDP per Capita" box, all data segments are valid except for the "GDP > $100 Thousand" segment. A screenshot of QlikView with the "Asia" region filter on is shown below. I can further filter the data by clicking on any other data filter segments that are white. As an example, if I select "$1 Thousand - $5 Thousand" in the "GDP per Capita" box and "40% - 60%" in the "Urban Population Percentage" box, the graphs and table update again. In this instance, the only countries that meet these filter requirements are China, Georgia and Mongolia. A QlikView screenshot with all three of the filters chosen is shown below.

Try it out yourself with the free trial

You can give Datamartist a try with this data, just signup and download the free trial, and then download a zip file will all the data, and the example Datamartist file.

Just extract all the files in the above ZIP file into the "My Datamartist" folder that the Datamartist trial will create when you run it, and open the "World Trading Example.DMC" file with Datamartist.

You'll find that Datamartist gives you a powerful, visual way to transform data from lots of places, and get it ready for great visualization tools like Qlikview in a step by step, clean, repeatable way.

On top of that, datamartist can be automated- so if you have data transformations you need to run on a schedule, you can design them in a graphical environment, test them, and then have them run automatically.

You know that all that data means something- but what?

One of my favorite code-busting stories is the epic victory over the Enigma code during the second world war. One of the reasons its of interest is that it was one of the early applications of computing- but the key lesson I think is from not the brute force computation done, but the strategies used to crack the code.

When you are trying to crack a code, one of the key things you need are "Cribs"- some way have samples of coded message and clear text. These cribs can radically reduce the number of possible ways a code can be decoded.

In the case of enigma, the allies would listen for German U-boat radio transmissions, while also using direction finding equipment to estimate their location. Standard procedure was for a U-Boat to first radio a weather report.

By painstakingly back tracking known weather conditions and locations of U-Boats when they transmitted it was possible to take advantage of that first weather report- there were only so many ways to say "Sunny and calm". Having this crib gave them a way to break into the code.

What is the point in terms of Data profiling? While it's critical to have the right tools to analyse the data (a data profiler like Datamartist, for example), its also important to get out there and talk to people, understand whats going on- collect some Cribs that will help it all make sense.

Once again, the U.S. Energy Information Administration was the source of the data (http://www.eia.doe.gov/cneaf/electricity/esr/esr_sum.html, Table 10). This time, the data was provided as one Microsoft Excel spreadsheet giving detailed information about the thousands of electric power generation companies in the U.S.A. The raw Excel file from the website had some merged cells, and had column names in multiple rows, so we had to clean it up a bit before importing. This is often the case, when using files that are more reports than raw data.

Even after these formating fixes, as is often the case, the data in its original form is not what we need- due to the large size and slightly different formatting of this spreadsheet, it was necessary to transform the data before combining it with the electricity consumer data. A visual map of these data transformations is provided in the screen shot of the Datamartist canvas below.

As seen in the screen shot above, the first data transformation carried out was a data summarizing operation. In this operation, the input data was summarized by state and class of ownership, converting the 3200 row input data file into a data file containing only 170 rows of data.

The next data transformation step required the use of the "Join" function. This is a very useful function in the Datamartist software because it allows two data files with different formatting to be joined together into one data file. In this instance, the summarized electricity generation data file was joined with a data file containing U.S.A. state abbreviations. This was done because the U.S.A. electricity generation input data file only contained state abbreviations. The result of this joining operation inserted a column containing the states full name based on the state abbreviation used in input data file.

With the join operation complete, the data was further transformed using the "Calculate" and "Sort" functions. The "Calculate" function was used to remove the data column containing the state abbreviations, as the previous "Join" function had added a column with the states full name. The "Sort" function was then used to sort the electricity data by state and class of ownership.

The data was then put through one last calculation function that renamed the column names, before being exported as a text file.
Now that the electricity generation data has been transformed, the Tableau visualizations can be used to present the data. It is worthwhile noting that since I joined the electricity consumption data with the electricity generation data before exporting it from Datamartist, I only needed to import one data file into Tableau Public 5.2. This single data file contained all of the information required to create both the electricity consumption and generation visualizations. As with part one of this blog post, I have included a Tableau dashboard summarizing state level electricity generation statistics below. This visualization is very similar to the visualization in the first part of this blog post. You can try it for yourself by clicking a state on the map and watching the table below the map update to present data about that particular state.

See the Data Transform Yourself!!!
All of the data files used in the data transformations in part one and part two of this blog post, as well as the Datamartist .dmc file, can be downloaded in a ZIP file here. You can see the data transformations discussed in these blog posts for yourself by downloading the free trial, install Datamartist, and then put all the files in the above zip file to the "My Datamartist" folder that datamartist creates in your "Documents" folder. Then just open the "Electricity Example.dmc" file in Datamartist and check it out.