It seems like everything just somehow gets fixed by putting it in the cloud. Security? No problem, our cloud does that. Data volume- The cloud is just HUGE man, HUGE. Dirty Data- Hey, clouds are made of WATER right? Trust us.

I goggled the word "Cloud" and discovered that Google places cloud computing over the boring water vapour hanging in the air type clouds. Although if you google "Types of Clouds" it does actually come up with a Wikipedia article talking about the different types of real clouds.

Then it struck me that there are probably different kinds of virtual clouds too, and we need to start naming those- so here goes a first run at it.

cirrus duplicatus

This cloud structure is particularly well suited to holding duplicate records. It consists of billions and billions of special data stores that implicitly avoid any sort of primary keys.

cirrocumulus delayious unconnectous

This cloud structure has a highly looped internal structure- data goes in, but will not attempt to come back out until it detects that your internet connection has gone down.

cirrus noworkus socialus

This cloud is used by all the big social networks- it enables lots of photo sharing, exchanging of comments, instant messaging, video viewing and has powerful anti-productivity filters in place at all times.

cirrus backupus disappontious

The backupus cloud promises complete data security- "In the cloud, everything is backed up ALL the time" - yet through a series of human errors (the cloud makes no mistakes- only its keepers) all data is uniquely stored on hard drives with failure modes that ensure no data will be recovered.

stratus brodcast allus privateous

This cloud has extremely powerful semantic filters that are able to detect the most embarrassing and sensitive data and then utilizes special high page rank domains to ensure your secrets are indexed immediately by all major search engines.

cirrius tellus momis

Just like "stratus brodcast allus privateous" only instead of using search engines it just emails the most compromising information directly to your mother.

This value profiling tool is just one of many of the Datamartist data profiling tools capabilities, download the free trial of the BETA to try all the functionality with your own data.

In some firms, their management reporting systems reflect the fact that as information technology began to be used extensively by business, often the very first functional area to be automated was accounting, and the first database within an enterprise was often the general ledger.

In many companies, the general ledger became the clearing house for all information- not just financial, and in effect became a data warehouse before the concept of data warehousing had even evolved.

The problem is, in some organisations, the data warehouse didn't come. The general ledger kept its place as the central repository for not just financial, but also management reporting. Finance argued successfully that the cost of all the business intelligence architecture was unnecessary- adding accounts and bolt on tables was cheaper. ERP vendors supported this by creating ever more flexible ledger structures, allowing additional ledgers for parallel accounting and management reporting.

Huge amounts of non-financial information is still stored in many general ledgers. There are so many reasons this is a bad idea. Here are just three:

1) It forces you to compromise on level of detail and drill down, and history

No general ledger can hold the level of detail available in many source systems. As a result, any interface from the sales system, manufacturing system etc. feeding into the GL will have to create journal entries that summarize a great deal of information.

While the detail of course will still exist in the source system, if your management reporting is all from a general ledger based system, upper management will tend to use this single source- and as a result important granularity may be lost to the decision making process.

This summarization also makes it more difficult to have drill down into the details, giving up some of the greatest benefits of modern business intelligence systems.

Finally, general ledger based data storage does not usually allow for the tracking of reference data changes over time. As sales regions are modified, and territories shift, comparing one period to another becomes increasingly difficult. Data warehouses, designed from the beginning to store this type of slowly changing reference information, can provide a much more insight and historical analysis.

The bottom line is, the data model of the general ledger module is just not designed for analysis.

2) It results in an overly complex chart of accounts and may even affect month end close

As the source systems become more and more capable of collecting data, the tendency is to want to increase the amount of management reporting. If this is being done in the general ledger, it means that further charts of account must be added, and an increased number of journal entries need to be done. Depending how the overall process is setup, its even possible that the increased complexity might affect the speed at which month end closing can be completed, if for no other reason that the same finance resources must both tend to the financial and the management reporting needs.

3) It discourages cross functional definitions and collaboration on analysis

By making one of the functional areas (finance) the center and owner of management reporting, a general ledger based reporting architecture can actually increase the severity of the information silos it is most likely trying to eliminate.

Because the general ledger reporting does not require all the detail available, each department only needs to provide the summarized information required by finance. While every department has to coordinate with finance, there is no requirement for departments to work with each other to coordinate data and definitions. While at a high level data is integrated, any benefit from more tightly integrating information across silos that a data warehouse can bring is lost.

In a very real way, a successful general ledger based management reporting system is in fact an impediment to progress for an enterprises business intelligence and data analysis evolution.

Because management reporting is available, the justification or need for a data warehouse is not felt as strongly. However, as needs continue to evolve, the effort expended in the constantly growing general ledger, and its impact on the financial processes, and the companies overall information management culture will become increasingly damaging.

Ironically, companies who failed to ever establish a general ledger based management reporting system could leapfrog their more financially focused competitors, as they embrace the modern data warehouse and the the tools available for data analysis.

A true data warehouse is not an easy road, and is only one component of a broader data analysis strategy.

Readers of this blog know that we advocate an approach that balances "Big Business Intelligence" with nimble, user focused data exploration and transformation.

In the short term, using the general ledger for management reporting can seem easier, but in the long term, it probably makes the task of creating an enterprise wide architecture harder- while your general ledger has been growing in the center, individual departments have probably been pursuing uncoordinated, fragmented business intelligence architectures of their own.

The currently released version 1.2.6 is of course also still available, but for those who don't mind risking a bug or two (we need your help in killing the last few) the Beta gives you a sneak peak at the new version and a bunch of new features that have been introduced to the Professional edition.

You can get it here. Whats in it? Lots of data profiling goodies;

Value distribution profiling

One of the important additions to the data profiling capabilities in Datamartist Pro is the value distribution explorer. This powerful functionality analyzes numeric fields and provides a value distribution graph (equal size buckets for row counts based on value) that lets you zoom in and out, and drill down into the rows to understand your data and spot any suspicious values at a glance.

Support for Regular expressions for pattern matching

The new version has added a very powerful function to the function library in the professional edition;

REGEX(text,regex expression)

With this function, it is now possible to use regular expressions (regex expressions) to evaluate if string values conform to desired data formats. Regular expressions are widely used for data quality testing, and there are lots of them available out there.

Custom data format rules

In V1.3.0 professional edition, it is now possible to add a series of custom rules for character mapping, creating much more flexibility in the data profiling tool. For example, its possible to map numbers, letters, punctuation characters, or any combination of them.

Data profiling block

In the professional edition, the category of data quality blocks has been added, and the first block released is the data profiler block.

This lets data profiling results themselves be used within your data canvas, as well as written out to files or database tables, with time stamps- enabling data profiling automation and tracking. It is now possible to define, calculate and track data profiling metrics automatically, sampling data at a sample frequency that lets you understand how your data quality is evolving.

We're excited about what can be done with this first data quality block- and we are sure our professional edition users will be tracking their data quality like never before.

I'll be blogging about all the new features in the coming days and weeks- I'm certain you'll find lots of things to like in the new version- give it a go.

As always with our betas, we love feedback, and we'll be giving some free PRO licenses away to our most active Beta testers.

Now, of course, standards are needed, and for a broad range of tools its counter productive (and horrifically expensive) to let everyone just use what they want. The cost of data centers, integration, etc. will be radically more if a company does not bring order to the chaos and the marginal advantages that a very specific or niche technology might have in one department is often obliterated by the increased support costs and integration issues globally.

But if a company looks at these savings that come from standardization, and extrapolates too far, they can fall off the other side of the benefits curve and find that they're hurting, not helping.

In the article in Harvard business review, the researchers also throw around the term "bitsmith" to describe someone who has both subject knowledge, and the ability to wrangle software, and to even create GASP! custom software that does what the team needs to get done. In many companies, current information technology dogma does not leave much room for people that have the time to be a "bitsmith".

In many companies "Custom software" is a four letter word. Well, I personally have used it, commissioned it and written it and know that often it can provide fantastic value- I've also seen people spend thousands of hours building something that never quite worked when an off the shelf tool a hundred times as good could be bought for a few thousand dollars. It's a matter of being realistic and learning how to see the difference between a problem that is specific to your industry/situation that could really benefit from some custom code, compared to a problem that is huge, main stream, and solved hundreds of times over by existing software vendors.

What I think can happen is a pendulum swing, where a company goes from "no standards, a jungle of wasteful custom software" to "Thou shalt use/buy only software on the following list."

The problem is, making a list that contains everything is just not possible. Things change. Stuff happens. It is possible that a single person might need a single piece of software that allows them to understand something, design something, communicate something that will make that software have a truly massive payback, justifying all sorts of pain and config on the part of technical resources and infrastructure.

The challenge, as always, is to have an open, working relationship between those entrusted with establishing and enforcing standards in terms of tools and those who are expected to use those tools to do business. As with so many things in governance, it's about balance, clear goals, and processes that allow for brilliance, change and creativity while not letting that process become the loop-hole that undermines all those savings standardization brings.

• Easy to install
• Easy to use
• ETL features and data profiling capability

Avoid using the wrong tool for the job

Enterprise ETL tools (Extract Transform and Load) are very powerful but often extremely difficult to use.

• expensive, particularly if multiple environments are needed
• require server infrastructure, configuration and setup.
• require expensive developers who have been trained in the specific programming language of each particular vendors tool.
• designed for performance and data volume, not ease of use.

Obviously they have their time and place, but when you want fast, visual access to your data, you end up getting slowed down by expensive ETL server overkill.

A better choice- the visual, clean ETL tool

Datamartist is designed to let you extract data from multiple sources, and then mix it, match it, transform it, and understand it.

It uses a visual block and connector model, with the concept of "Data canvases" that let you easily manage and simplify complex data transformations. But unlike many overly complex ETL tools, Datamartist provides visual, configurable blocks, rather than requiring code.

Easy to install

Datamartist installs in minutes, and runs on your desktop, giving you control of your data, and what you need to do. Don't configure servers, don't worry about installing the right version of Java, don't spend hours searching wikis and forums and tweaking config files. Just download it, single step install it, and use it.

It makes it easy for you to take a snapshot of the data you need- locally with cut and paste or drag and drop from files, and locally or remotely with native connections to SQL Server, Oracle, MySql and MS Access, and pretty much anything else via ODBC.

And since the Datamartist data transformation engine can be run from the command line or scripted, it can also be automated to implement ETL tasks running on a windows server.

Speed up data delivery, reduce cost.

Datamartist provides a flexible, simple to use ETL environment that will let you shorten your time to delivery significantly for a wide range of data transformation tasks.

• Deliver small and medium sized data transformation tasks more quickly
• Build rapid prototypes and proofs of concepts
• Automate data profiling and data quality monitoring

Give the Datamartist ETL Tool a try

You can download the Datamartist trial and be up and running in minutes. You don't even have to register- and you will have full access to a fully functioning version of Datamartist to try out this simple, visual ETL tool on your own data.

We're also very excited about V1.3.0, currently in private beta. If you'd like to participate in the public beta, drop me a line at "beta at datamartist.com", and we'll send you a link when that download is available.

"Using the crayon takes too long and sometimes I make mistakes." Then he paused for a second. "But if it's something different- then I have to draw it. No stickers for that."

And off he went, blending hand drawn custom crayon work with high speed sticker application.

It strikes me that what my son has figured out as a basic rule of thumb in arts and crafts for the use of stickers, is a pretty good analogy for design of data entry systems.

Whenever you can, use something that restricts the users choices to a fixed, understood set of responses. Use pre-made data stickers.

The enemy of data quality everywhere is the gaping, un-validated free form text entry field. Only linguists and unstructured text analysts can get excited about the "endless possibilities" of what your users and customers can enter into those fields.

We've all seen the horrors of names and addresses run amok- "John A Smith", "Jon A. Smith", "John Smith Jr.", "Smith, John A" or the even more amazing "John Smith (new customer)".

If you're in data, you don't want endless possibilities. You want ordered sets of data that conform strictly to well defined rules. Eliminating duplicates is a complex and time consuming effort. Stopping as many of them before they are created is the first, best thing you can do to get a handle on the problem.

So think stickers. For every field ask yourself- can I make this a combo box? radio buttons? Can I do auto search in the existing records to suggest close matches? Anything to stop users or customers from making things up- and to have the data points they enter conform to a defined domain.

The more constrained a field is, the better the chances are that the data stored in it will be useful... unless of course you make it so constrained that you force data quality to suffer.

There is such a thing as too much...

Every good rule has its exceptions, and the evil side of overly constraining your data entry folks is that because they are smarter than computers, they'll find ways to invent entirely new encoding methods.

If you tighten the entry on the postal code too much, so that international postal codes won't fit, you can be sure that data entry clerks will discover that by entering their own postal code, and putting the customers postal code in the comment field, they can get the system to accept the record (and at least feel as if they had tried their best to get the data needed in there).

This is where which stickers you have in your collection starts to matter.

Have you ever noticed that at well run events, they always have some blank name tags, as well of the pre-printed ones? That and a magic marker makes sure the process can go on.

In the end, you'll need to balance between the two extremes. Tighten up your data entry and interfaces as much as you can, but realize that there is a point of diminishing returns, and in fact probably even a point where your data totalitarianism will be hurting your data quality, not helping it.

Now of course, there are some pretty high end tools that let you create all sorts of rules, and others that let you comb through the data and cleanse it, checking those postal codes to states and cities, and doing all sorts of fancy matching and analysis. There is definitely an important role in many organisations and systems for approaches and tools such as these.

Using data profiling tools like Datamartist will help you understand what issues are making it through your defenses.

But if you are not doing it already, focusing on the point of entry with practical, balanced techniques will make a step change improvement to your data quality.

Data integration can be like a really well put together pizza- lots of good cross-referencing cheese-data to keep everything in its place, great crust that supports it all, and a universal appeal that might even get people to try something they wouldn't normally consume (data wise).

But without data quality, data integration can make pizza that nobody really wants to eat, and rather than enhancing the value of your data, your data integration efforts can make your bad data even less consumable than it was on its own.

While combining data from multiple systems can generate huge insights, it is important to understand that moving it and combining it with data from other systems will not always increase its value.

With good quality data you can have fantastic results, but bad quality data requires so much effort and transformation that often your payback on doing the integration will be non-existent.

Data integration enthusiasm

So what happens when an enterprise hears its stomach rumble, and starts thinking data pizza?

Enthusiastic analysts spring into action, building various mockups of all the fantastic dashboards that they will be able to produce, once the data integration is done. Terms like "near-real time, balanced, cross-functional score cards" start to get bounced around, and pretty soon, budget proposals and appropriation requests are flying from color printers everywhere.

Whats unfortunate in many cases is that cooler heads don't stop to ask the question- "So... all this data we are going to put together, is it any good?"

When you are making your pizza, you have to know if the cheese has been left out a bit too long or the green pepper is soggy.

What can be worse, is that if heroic measures are taken to try to get the data to fit together, the integration jobs themselves might actually degrade the data quality further- or eliminate levels of detail that are not compatible, actually hiding important trends and structures. A risk of integrated dashboards is that they pander to the lowest common denominator.

So if you are planning to do some data integration, to build a data pizza, think twice about putting that moldy pepperoni from the CRM system on it- sometimes less is more.

In fact, it might be that data integration is not your first concern- improving the quality of the data in all those data silos will actually improve day to day operations immediately- and make any future data integration project cheaper, and more successful.

Any great chef will tell you- no matter how complex the recipe, and how impressive your kitchen and equipment, the raw ingredients matter.

But you don`t allow them to come see your property, and your sketch doesn't specify anything about the existing landscaping- just the final configuration. Do you think the landscapers would be willing to offer a reasonable price ?

Unlikely. What if there are existing patio stones to remove- or an in-ground swimming pool that`s got to go?

No landscaper would take on a job without understanding the lay of the land, and the existing conditions. It would be impossible to estimate the job. Anyone who did would give you a huge price to cover themselves, or demand extras upon discovering the extra work.

Yet when companies hire consultants to build them business intelligence solutions, or do data migration, it often happens with only the roughest outline of the existing data sets. Certainly, often a data model is included- but knowing what the table SHOULD contain rather than what it does is just not the same thing. It never ceases to amaze me that the simple, cost effective practice of data profiling is just often not part of the initial phases of so many business intelligence and data migration projects.

With the right data profiling tool, and just a few days work, its possible to gain a huge amount of insight into the data quality in your systems, and as a result, be able to make radically more accurate estimates of the cost to go from the "as is" to the "to be".

Phil Simon talked about this in a great post on the Data flux blog called "What Consultants Don't tell you", and raises an important and somewhat ugly truth- many times, service providers don't WANT to do data profiling because it reveals the true extent of the work to be done, increasing the budget requirement, and makes the project less likely to be approved.

Now certainly, we can't use a broad brush to paint all consultants, but it does lead to a reduction in the number of times valuable tools such as data profiling are recommended even though in my opinion they are a low cost, no-brainer, do it unless you are crazy first step to any major project.

You are going to spend potentially millions of dollars on a business intelligence or data migration project- spend a few weeks to look at the data with the right tools first for goodness sake!

If you want to get a reasonable cost estimate, and you want to go into your business intelligence or data migration project with open eyes, don't imagine you can know what it will cost to get from here to there if you don't take a good look at where here really is.

Full disclosure- of course, you are reading the Datamartist blog, and Datamartist has lots of data profiling functionality- so you have to understand that we are incredibly biased on this topic. If you are able to overlook our inherent bias, give the tool a try- you`ll discover things about your data you might not have wanted to know, but its better to face the truth prepared, than to rely on wishful thinking, and then discover the bad news when you're well into the project, and your budget is almost gone.

It's hard because although methods like data profiling can identify data quality problems, exactly who is to "blame" and how to manage it is difficult.

In thinking about this a bit more, I realised that the discussion was all about sticks- and not much about carrots. We discussed different ideas about how to proportion cost out, (which makes sense as the swear jar is about putting money IN, or punishing the offender).

What about working it the other way around? By using automated data profiling, and making the metrics now time based and available we could track data quality in key data sets, both in absolute terms (numbers of rows with issues), and relative ones (percentage of customer records with problems, etc.). This would allow you to have data quality dashboards.

It would then be possible to establish a bonus plan based on data quality. It could pay out for improvements (as a percentage), or have certain reoccurring payments that would decrease and then stop if data quality fell below a target level.

While it is still necessary to identify who is responsible for which data set's data quality, and as with all reward schemes the people responsible must also have the means to improve data quality and therefore reap the reward, I think in a number of situations this would be possible- for example, data entry clerks would undoubtedly double check each address more carefully if they knew there was a tangible reward to do so.

It seems that two key things are necessary to make this kind of bonus plan work- first, the ability to automate data profiling, and have meaningful metrics that can't be "gamed" (because if there is a way, people will find it), and secondly to be able to identify the savings due to improvements in data quality- because that's whats funding the bonus pool, after all.

Have any readers implemented or heard of such a plan? Do you track data quality using an automated data profiling tool and data quality dashboards?