Sometimes an analyst needs to take off the accountants hat, forget the urge to chase down every last penny, and instead put on their surfing gear, grab the data surf board (i.e. their set of prefered data tools), and just surf some data.

There are some cases were "Exact" is the only acceptable level of data quality. When we're sending invoices to our customers, not only does the amount need to be right to the penny, the invoice needs to make it to the right person, on time.

But sending accurate invoices is not exactly the cutting edge in terms of data.

The challenges for todays data driven organizations are to be able to make sense out of the ocean of data available, and to do it faster than the competition.

And the ocean does have the highest quality water all the time. In fact, some of the data is downright dirty.

But there is a lot of it, and it can tell us things.

Just like ocean surfing, when you are data surfing you can sometimes ride the wrong wave and end up underwater, you can spend hours on your board in flat water, hoping the surf will come up but instead getting no where.

But when the wave comes, and you ride it, letting yourself go with the flow- while it won't give you any exact answers, it will give you a "feel" and sense of where the wave is going, and if you are fast to grab it, and your board is good, you might just find yourself getting some awesome, gnarly insight that will score you some competitive advantage- and you just can't surf that wave if you are worried about only looking at Exact, perfect, "we've checked it three times" data.

Data Surf is up- and by the look of the ocean it's going to be a rocking rolling ride- good luck out there!

In the vast majority of cases, useful data sets are not static, but are being updated, added to and purged constantly.

Data quality monitoring aims to provide data quality information that is also being constantly updated, and can be used to detect issues quickly, before the bad data piles up.

Don't let those bad records pile up.

Imagine a company that does a mailing to its customers every 3 months. Imagine that a new call center training program is put in place, and unknown to all, customer information starts being incorrectly entered and updated due to an error in the program. When do you want to know about the growing number of invalid customer records? When its time to do the mailing, after 90 days of bad data generation, or after just a few days of problem entries?

ALERT! ALERT! Bad data Alert!

The trick to data quality monitoring is having a set of data quality rules, and profiling tests that will automatically give indications of issues. Some rules are easier than others, but the idea is that each record or group of records in a the table(s) in question is checked against a series of tests, and the number of data quality rule infringements are tracked. As the size of the table grows, if the overall percentage of bad to good is increasing, you know you have an issue. If it spikes up, you sound the alarm.

Actually having a large fog horn wired up in the CEOs office is optional, due to the challenges of false positives (detecting a "Bad" record when in fact the record is ok).

I would also be very cautious in having automatic data modification processes going on, but setting alerts that will notify those responsible for potential data quality issues is a relatively straight forward exercise and will at the very least improve your visibility of data quality trends. What you do to deal with them is up to you, but at least some of the battle is being aware of the problem.

It monitors a number of things, but the key here is to make a decision for ever record if it is "Good" or "Bad", and give some analysis of which data quality rules are broken. You can see in the upper right, there are row counts of infractions by a number of data quality rules. In the middle is the overall Bad vs Good split on the records.

The datamartist Pro version can quite easily create automated data quality monitoring. Using visual blocks, and arranging them on a Canvas, Datamartist lets you create data quality rules, define profiling on selected columns, and then automaticaly, as a scheduled process, place the results either in Excel reports/dashboards or into a database for use by your reporting tool of choice.

By understanding how your data quality evolves, and catching problems early, you can reduce the amount of cleansing you need to do, and communicate clearly and regularly to your organisation the progress your data quality programs are making.

Tableau is a wonderful, powerful visualization tool. If you have the data, it will generate insightful, powerful dashboards and reports.

But actually getting the data is often the tricky bit. And having it appear in your dashboards and reports without having to press lots of buttons is the goal. (No one likes to have to do a long series of cut and paste operations to get the data in place).

Datamartist is a tool that lets you combine data from multiple sources, format it, clean it, organize it, and then make it ready for tools like Tableau.

Because Datamartist can be automated from the command line, and commands on Tableau server can be run from the command line using tabcmd, it means that with Datamartist and tableau together, you can have a complete and automated reporting solution.

Have you ever wished that you could update your Tableau dashboards automatically every day? Do you have non-database information in the form of spreadsheets that you need to integrate into your Tableau workbooks?

With Datamartist and Tableau together, you can automate all these tasks, integrating formal database data with spreadsheet data from shared drives seamlessly, and quickly. And when your users log into Tableau, they see the dashboards they need to see.

Datamartist lets you import data from Databases, excel files and text files, define data transformations that prepare and clean the data for visualization, and then export it to a location (very often a database, but potentially an Excel workbook or file) where it can be easily, automatically read by a visualization tool like Tableau server.

You can learn more about what Datamartist does here.

To give you an idea about what a batch file that first runs a datamartist canvas, (which loads data from multiple sources, transforms it and makes it ready for Tableau) then refreshes workbooks on a remote Tableau Server automatically, here is an example of a bit of simple batch file code:

The first line runs Datamartist- this runs a .DMC file that you have built with datamartist that can extract the data from multiple sources- databases, files, internal hard coded sets- then do joins, segmentations, filters, calculations etc. and export the data to the location that the Tableau server workbooks are connected to.

"C:\Program Files (x86)\nModal Solutions Inc\Datamartist\Datamartist.exe" RUN /f "C:\MyDataMartistFile.DMC" /l "C:\WhereIWantTheLogFiles\

Then, using a utility called tabcmd from Tableau, you can log into a tableau server, and refresh the workbook, thus updating all your tableau dashboards to do this, you use the following lines in the batch file:

"C:\Program Files (x86)\Tableau\Tableau Server\7.0\extras\Command Line Utility\tabcmd.exe" login -s MyServerURL -u MyUserName -p MyPassword -t MySite
"C:\Program Files (x86)\Tableau\Tableau Server\7.0\extras\Command Line Utility\tabcmd.exe" refreshextracts --project "MyProject" --workbook "MyWorkbook" --site MySite

You can find more information about tableau's tabcmd on their site.

Net result? You have full automation from data to visualization, Datamartist takes care of the data, Tableau creates fantastic dashboards.

If you build a batch file, and set it to run once a day, every morning when your users log into Tableau server, they'll see updated data, ready to go.

Try Datamartist- its a full function free trial. Find out easy it is to automate tableau server with sophisticated data extraction.

There is an ongoing debate in Business intelligence and Analytics circles about what role spreadsheets have in data management.

At the extreme pro-spreadsheet side, Excel can fix all, be all, and IT departments should just be disbanded, because the world can run on workbooks and macros.

At the other end of the spectrum, all that is evil and wrong, all the corrupts data quality, all that is unholy in the realm of data management process and master data management can be traced back to renegade spreadsheets, the "culture of excel" and spreadsheet applications should be uninstalled from every desktop never to be seen again.

And of course, in the middle, in the real world, we all use excel constantly, every day, all the time for lots of things.

Some organisations do use it too much, others probably spend way to much money on their business intelligence solutions because they don't let their users use it enough (although fewer of these than the first sort is my guess).

What is my take on it? Excel is a fantastic scratchpad, and a powerful reporting tool. It is not, however, a data repository. The official version of the data needs to be in a database, managed by someone who knows data, data models, and master data management.

But if you are looking for a quick tool to create dashboards, reports, and give people easy access to data- don't discount Excel.

This is why we've recently greatly enhanced Datamartist's ability to read and (more importantly) generate Excel workbooks. The fact is that excel is an application that is installed on every machine, is familiar to many, and is an excellent way to deliver data. Not store it, but deliver it.

The challenge that many Datawarehouse projects have faced is that they are competing with all the excel spreadsheets that circulate through the typical organisation. The business intelligence team is hard pressed to deliver new reports as quickly as they are demanded- and as a result people bypass the system and create a web of spreadsheets that become the "real values".

How can you get data out of databases and into excel? How can you export from SQL Server to excel, or oracle to excel, quickly and easily? Datamartist provides a way to build reusable, automated queries and transformations from multiple databases and files and to generate excel workbooks with graphs and tables in an automated way.

You can run Datamartist as a scheduled job, generating the excel workbooks you need regularly, so users come to understand that while the spreadsheet is useful, it is ephemeral. The master copy of the data is in the datawarehouse. Excel workbooks are reports. But because they are available quickly, a hybrid solution of the core data warehouse and excel reports can deliver better data in a format users are comfortable with.

What are some of the things excel does particularly well?

• Good graphs, flexible, easy to set up and format
• Excellent pixel by pixel formating control- ability to just get what you want in terms of look and feel- add in images, tables, anything
• Great integration with MS office (obviously) so going into Word or Power Point is easy
• Great control of printing/formatting- ability to generate PDF by installing a print to PDF capability
• Incredible adoption and broad compatibility- chances are, whoever you want to share with has excel.

So don't be an extremist on either end of the spectrum- don't think for a moment that "just spreadsheets" will solve your data management needs- on the other hand, don't be an excel bigot- excel has a roll to play, and in fact, can be a tool to drive adoption of your data warehouse and master data management process.

Being able to pivot and transpose data is a key part of reporting and data analysis. With the recently released V1.6 of Datamartist, there is new functionality in the Pivot block that lets you turn a data set that looks like this:

Into one that looks like this:

The columns have been transposed to rows, and the former column name is now a value in each row.

Why would you want to do this? One very common reason is to prepare data for olap tools such as Tableau or Qlikview- Having each individual country as a column will not allow you to take advantage of their slice and dice capabilities, with Country as a column, and the country names stored in each row, you can take advantage of pivot tables and olap tools.

How its done in Datamartist

To do this in datamartist, we use a PIVOT block, and connect it to the data set we want to transpose:

Then we simply specify which columns are to be transposed, and provide a name for the the two new columns- one column will contain the former column names, the other column the values.

You can try this yourself with the free trial of datamartist- you will find you can import data from files, excel and databases, and using easy to use blocks, create powerful data transformations, making your data ready for use in great tools like Excel, Tableau and Qlikview.

We are pleased to announce the release of Datamartist V1.6, and a powerful new capability for the generation of Excel based dashboards.

With the new version, Datamartist can insert data sets easily and visually into Excel workbooks as well as having the capability of defining a template, allowing automation of dashboard generation.

As always, data transformations are created visually, using blocks and connectors- in this case, we are analyzing some sales and profitability data.

We use blocks to summarize, transform and pivot our data, and then export it into specific areas of the desired excel template.

By inserting the data onto a "data" sheet in the template that has links to charts or tables on the dashboard sheet, we can create powerful, clear dashboards.

When in datamartist, you can see clearly where the data sets are going to be inserted- and datamartist generates the excel workbook(s) needed with all the data inserted. Because Datamartist can be run from the command line, you can automate your excel reporting, pulling data from databases and files, and generating professional looking dashboards that are easy to share and use.

The Excel export capability is far from the only enhancement that V1.6 brings. I'll be blogging, and providing examples of new functionality in the coming weeks.

Well, it's the time of making and breaking resolutions, a time when setting realistic goals is sometimes hard to do with all the optimism of the new year.

Sometimes, we make decisions NOT to set a goal, because we don't want to break it.

You might be thinking you really should step up your data quality monitoring- get some data profiling underway to help identify the data domains and areas you most want to tackle in 2012. But you might be also thinking that with all the pressures and cutbacks that many companies are facing, you don't have the resources to implement a full scale profiling and monitoring effort, and so might decide to delay.

Don't wait. Just do it. The perfect is the enemy of the good.

Rather than worrying about how much of your data you are going to be able to cover, or that you can't devote enough resources to tackle all of your reference areas at once, work at the problem from another direction.

First, start with master data.

Master data is the data that all your other data is made from. It's the data everyone uses to view the massive piles of transactional data, so one bad row in a master data table, and the impact is felt across perhaps hundreds of reports, and multiple time periods. If you have a product in the wrong category, then every transaction, across perhaps hundreds of customers, and all time, will be mis-catagorized, and every total, sub-total and calculated metric using it will suffer.

While bad transactions are bad, bad reference data is deadly. Bad reference data takes a good transaction and messes it up.

Worst first!

Make a list of your reference tables/area. Customer, Product, Chart of account, etc. etc. What are the most important for your business? This isn't something I can tell you- you have to think about what is most critical.

If you are a company that purchases large amounts of materials from many vendors, and purchasing decisions are fast paced and critical, then maybe it's your vendor master, and your accounts payable.

On the other hand, if you have lots of interaction with your customers, and errors in the customer master cost you business, then start with that.

The key is to first make the list, and then think to yourself "if I have bad quality data, where am I most afraid it will be?" Start profiling there. You want to find the worst first, and fixing that will have the greatest positive impact.

Get to know your data

Don't worry about setting complex or work intensive goals right away. Data profiling is about data discovery sometimes. You need to wade into your reference data, play with it, tease out patterns and relationships. As you get to know your data, you will be able to better identify where there are issues to tackle, and where root causes might lie for data quality issues.

One approach might be to simply resolve to spend an hour a week, every week, profiling some data. If you aren't do that now, you will find that even just a bit of time set aside will give huge insight- sometimes we get too busy to do the basics, and we miss opportunities to make significant improvements with relatively little effort in our data.

We are pleased to announce that Datamartist V1.5 is now available.

This version of Datamartist brings with it some useful new functionality, including new functions that can be used in expressions, new capabilities in terms of exporting to databases, and a new data block.

In this post, we'll look at two new features, the Pivot block, and the enhanced database export capabilities.

Pivot Block

Our beta testers loved this new block. The pivot block lets you do the equivalent of a cross-tab query, rolling up a measure, and distributing the value in a new set of columns, where the column names are provided by the input data set.

Here is a simple example showing how it works:

Say we start with a set of data that has mutiple rows for each date, and different values in the color field, and a quantity measure:

Then we can connect one of the new pivot blocks to this data set like so:

The pivot block lets us select which columns to include (this defines the level of detail to roll up to), which string column to use to generate the new column names, and which measure to use as well as the rollup method (sum, average, min, max)

The result? The output of the pivot block looks like this: now we have a summary by color for each date, with a column for each color value.

Database export enhancements

Now, when exporting to a database, there are a number of new options.

One of the most interesting is the capability to execute SQL commands in the database either before and/or after the data is exported into the table.

This provides the capability of running stored procedures, or launching follow on database side processing after Datamartist writes the data into the DB.

This is a powerful new capability, and makes it even easier to integrate datamartist into various systems, and get your data quality and profiling data where you need it.

If you haven't checked out datamartist yet, we're not sure what you are waiting for- download the free trial, and give it a go.

I've had lots of fun over the years building the little cartoons that have become a regular feature. Here are a few, reposted together just for fun.

Data quality super powers.

Fighting the anti-data forces of evil.

Data silo fun.

The joys of a moving target

Data migration tools.

What's the difference between good data and bad data? It is much like the difference between good children and bad children- the bad data doesn't follow the rules.

But what are the rules? Unlike the rules for kids, which have been fixed in stone for decades (or at least, parents wish it were so), the rules for data are slippery things that depend very much on the context and the database.

While it's a complex subject, some basic rules of thumb can avoid the deeper rabbit holes.

The first thing to understand about Data Quality rules is they aren't as easy as they may look. Data is in theory something in the ordered world of computers, but in reality is in the "flexible" world of humans. A huge amount of data is entered by members of the group "Homo sapiens" (or mutilated by software written by members of that group) and as a result is not as ordered as we would all like.

The challenge for data quality practitioners is to remove the chaos injected by those highly involved primates (us) and make the data the sterile, ordered, never any question about anything type that we all imagine in our fantasies.

But how?

In the end, it is amazing how powerful and complex the various solutions to this problem are.

But I suggest that there are some basic principles that can help guide us.

First- do no harm.

One of the risks of any data quality initiative is that it actually screws up the data more. Don't define rules that are so complex, and so sure of themselves that they actually make the data worse. Be humble. Don't change data unless you are pretty sure it's a good idea. Err on the side of not screwing up the original. And keep a copy of the original- so if things do go off the rails you can undo- or at least try to understand what when wrong.

Go out and talk to the people

Don't sit in your ivory tower and speculate as to what the data means. Go out there and watch people enter it in. See what real world type things are happening that never make it into bits and bytes.

Attack the basics first

Focus your first efforts on dealing with the basics- they will resolve the vast majority of the issues- don't chase after the outliers until you have the "easy" cases taken care of- the tough stuff is a case of diminishing returns- look first at how to fix processes and train your people to make the majority of typical data entry cases more accurate before you start looking into artificial intelligence based hyper-multi-semantic-algorithmic-learning-matching-holistic-flux-capacitor data quality systems.

Less is more- the fewer rules the better.

So whats the rule about making rules? Try to make less rules, and test them in a pragmatic way. It is possible to have so many rules that the rules themselves have data quality issues- don't go there.

Sometimes the simplest things will bring the greatest benefit.

In the coming weeks, I'll be posting about how to design, implement and monitor Data quality rules using the Datamartist tool.