Data integration can be like a really well put together pizza- lots of good cross-referencing cheese-data to keep everything in its place, great crust that supports it all, and a universal appeal that might even get people to try something they wouldn't normally consume (data wise).

But without data quality, data integration can make pizza that nobody really wants to eat, and rather than enhancing the value of your data, your data integration efforts can make your bad data even less consumable than it was on its own.

While combining data from multiple systems can generate huge insights, it is important to understand that moving it and combining it with data from other systems will not always increase its value.

With good quality data you can have fantastic results, but bad quality data requires so much effort and transformation that often your payback on doing the integration will be non-existent.

Data integration enthusiasm

So what happens when an enterprise hears its stomach rumble, and starts thinking data pizza?

Enthusiastic analysts spring into action, building various mockups of all the fantastic dashboards that they will be able to produce, once the data integration is done. Terms like "near-real time, balanced, cross-functional score cards" start to get bounced around, and pretty soon, budget proposals and appropriation requests are flying from color printers everywhere.

Whats unfortunate in many cases is that cooler heads don't stop to ask the question- "So... all this data we are going to put together, is it any good?"

When you are making your pizza, you have to know if the cheese has been left out a bit too long or the green pepper is soggy.

What can be worse, is that if heroic measures are taken to try to get the data to fit together, the integration jobs themselves might actually degrade the data quality further- or eliminate levels of detail that are not compatible, actually hiding important trends and structures. A risk of integrated dashboards is that they pander to the lowest common denominator.

So if you are planning to do some data integration, to build a data pizza, think twice about putting that moldy pepperoni from the CRM system on it- sometimes less is more.

In fact, it might be that data integration is not your first concern- improving the quality of the data in all those data silos will actually improve day to day operations immediately- and make any future data integration project cheaper, and more successful.

Any great chef will tell you- no matter how complex the recipe, and how impressive your kitchen and equipment, the raw ingredients matter.

In this version, we've introduced a Standard and Pro edition, letting customers get the features they need at the right price.

• Datamartist Standard: $349
• Datamartist Professional: $745

A comparison of the feature sets explains the details.

Whats new in V1.2

Data source import enhancements

• Ability to cut and paste between Excel, Text files, the Datamartist canvas and any Datamartist data viewer.
• New integrated data source repository with drag and drop to canvas.
• SQL Editor to allow the creation of SQL queries to get data from databases.

Running Datamartist canvases automatically

Now that Datamartist can be run from the command line, it is possible to schedule datamartist transforms- even running it on a Windows server. Details about the logging and options are here.

Edit Internal data sets.

The addition of fully editable internal data sets that are stored within the DMC file itself gives a powerful new ability to create "What if" type scenarios. Imagine you want to see the effect of changing the sales regions slightly- just copy and paste the existing from a data viewer onto the canvas- that gives you an internal data set block with that data in it- now you can add a column "New Region" or rename the column, then edit some values, join it back into the original data with a join block, and be trying different scenarios in no time.

We're excited about this new release, and thanks to all our customers and testers for their feedback- we're glad to be incorporating some of those great ideas into the product.

If you haven't tried Datamartist yet, this is the perfect time, and now with two editions to choose from you can get the features you need at the right price.

I believe a key to understanding how information moves around your organization is to think of it as a mini economy. (I know, the economy is not our favorite subject right now, but bear with me).

There are information suppliers, and information consumers. The consumers are willing to pay more or less for different types of information, and different methods of supplying information have different costs. In the end, the market decides what gets done and what does not get done.

And like many markets, there is also a underground economy- places consumers go if the official prices don't make sense, or the products they want are not available on the open market.

In many companies, the IT department in theory has a monopoly on information supply, however the underground is active and constitutes a significant supply. The underground in this case is all the excel spreadsheets, the MS Access databases etc. used to make the shadow systems and spread marts. Spreadmarts seem to exist in the majority of enterprises- I've mentioned an interesting study regarding these shadow systems previously, and the attitudes people have.

To help illustrate this I am going to make up some data and put it in colorful graphs.

Looking at the first graph, in broad terms a data warehouse based approach will have higher costs than one based on data marts (because data warehouse provide more cross enterprise integration, which requires more effort), and the spreadmarts will have the lowest perceived cost. It's important to note that the actual cost of spreadmarts are higher, but percieved cost is what drives the consumers choice.

The trick is that because the percieved cost of spreadmarts is so low, and because there is no sanctioned enterprise solution to compete, a significant amount of effort is put in to these systems for any type of analysis that is percieved to be possible. Of course for certain data volumes or complexities there is no alternative to a full fledged data warehouse or data mart project, but for almost everything else, business users and analysts will often try to go it alone creating a chaos of spreadsheets and data bases.

The problem is, even "experts" can't accurately estimate how much effort the data analysis is. So estimates for how long it will take to "whip it up in excel" by non-experts are almost always low by orders of magnitude.

Don't dictate. Engage with sanctioned tools that work the way people want to work.

The key to adjusting this market imbalance is to introduce a new sanctioned product line, in effect undercutting the "black market".

This is exactly what self serve data transformation is about. Rather than leaving users to do it themselves in Excel- IT can provide specific tools, and thereby reduce the amount of completely opaque data transformation going on, while still providing users with the ability to get what they need.

So why is that better?

• It opens up the dialog - Talking is better than having a "Us" vs "Them" mentality. It lets you meet the people involved, lets you discuss their challenges with them, and provides an opening for discussion of important topics like data quality, master data management and data security.
• You'll know who the power users are - Right now, it is potentially anyone who has Excel- chances are that's everyone in your organisation.
• It gives you visibility on what matters to the business - If you know what the hot topics are, it can help you keep the official systems relevant and prioritize your efforts where they will do the most good.

What has to be different in this new relationship, however, is that IT has to understand about the "self" in self-serve. People will do things that no self-respecting ETL developer or data warehouse architect would ever sanction. If you clamp down and stop them, they will abandon the tools and return to the wild west. IT believes that it has the power in the relationship, but in fact the users are able to walk at any time. So add value, communicate, educate, but don't dictate. If your relationship with the business users, and the "Kings of the spreadmart" is poor to start, you have to give it time to evolve.

"But we just can't let them do that."

Resist the urge to clamp down.

Keep your systems secure, guard your infrastructure, but don't have any illusions that you can stop people from analyzing and transforming their data.

If they want to calculate net sales in a particular way then they'll do it in excel, and it will be the number that the CEO sees. The business is made up of grownups, after all. IT has a responsibility to explain the issues and challenges that shadow systems and rampant spreadsheeting can cause, but I have yet to see or hear of a company where an authoritarian approach works. As Princess Leia said- "The more you tighten your grip, Tarkin, the more star systems will slip through your fingers."

Arming the rebels

The business intelligence vendors are all realizing what the crowd pleasers are- really good integration into office applications, excel at the forefront. People want at their data.

Microsoft has of course long provided the main weapons for the shadow systems, MS Excel and MS Access- and they are going nuclear with the addition of "Power Pivot" to Excel 2010- although it is largely a presentation layer tool, and probably won't be used widely for data transformation itself.

Trying to fight all this with the standard tools of closing down the ability to export data, hiring an army of report writers, and constantly raving about the dangers and pitfalls of run away spreadsheets is like pushing on a rope.

Provide a safe, legal alternative to the free for all.

Talk to your business users. Understand their needs. Provide them with tools. Work with them to both empower responsible analysts, and avoid the worst issues that existing shadow systems are creating.

I believe in an approach that firmly places people in the center of a new way of looking at the data analysis process.

In the past, “big business intelligence” created an infrastructure heavy, highly centralised and technology focused approach to getting data from source systems into reports in the hands of the users. Under this regime, users were not to be trusted with raw data, but were given tightly controlled, managed and aggregated reports in order to protect the “single version of the truth”.

The theory and practice were tightly defined, and had been honed over decades of business intelligence and data warehouse orthodoxy. Giving raw data to end users would lead to chaos. Letting end users define new ways to look at the data would corrupt the master data, and lead to everyone looking at something different.

You can guess the sort of response this "don't give them the raw data" approach gets from capable, curious people that want to get down to some real analysis.

But to be fair you can see why these concerns are thought to be well founded. Almost every large enterprise is awash in a sea of excel files and a tangle of links and formulas. Excel is a wonderful tool, but it only offers the illusion of solving the data transformation problem. It is a much better reporting/dashboard tool than an ETL. (Although in the right hands it can do remarkable things.)

And this is the true state of affairs now. When the “official” system does not provide the answers that the business needs the people who need to make decisions get the data anyway, and they do it themselves. They do it in excel, they take night courses in Structured Query Language (SQL) they hire consultants (or even summer students) to build rogue data bases that they run on servers hidden under desks to get at the answers they need.

It is easy for the data warehouse theorists to highlight the clear issues with "spreadmarts" and "shadow systems".

But we need to be pragmatic. The reality of building a centralized structure that imposes strict formal rules and change management processes is that often while it does ensure that there is only one version of the truth, it is a version of the truth that no one can use because it has been so formalized, aggregated, compromised and delayed that by the time it is delivered the pressing business questions have changed and meaning has been expunged. The data warehouse becomes reporting rather than analysis.

Its clear that enterprises need this kind of reporting- I'm not advocating abandoning the existing approach- but augmenting it. Up till now, the solution has often been "more of the same".

The regime decided that the solution was to add more technology to the central systems, increase enforcement, and search out and repress all the dissident data manipulators. The data resistance was forced to go underground, to hide their spreadsheets, to outwardly appear to be following the official line.

It is very true that there are some risks in allowing people to analyze their own data, but there is also a reward. There are a small group of people who love data, who understand the business questions, who work to tease insight out of a steaming pile of raw data and can find things that are game changing. Massive, formal, designed by committee data warehouses can deliver a powerful and useful view of things, but they rarely offer flashes of insight. When they do, it is often during the design and discovery process- rarely by users using the system after it has gone live.

The Datamartist tool has been built based on the belief that both formal, centralized systems AND local, personal data transformation have a place in the architecture and that both should be official places.

People can be trusted with the data. In fact I think for an organisation to truly be successful at mastering its information, they have to be.

We have to realize that we can't allow our obsession with the quest for a single version of the truth to turn us into totalitarian regimes, certain that OUR truth is THE truth, and that messing around with the data is by its very nature subversive and dangerous.

Data to the people.

Microsoft Access has a clear role to play when a small, light database application is required. However, it has a learning curve, and is not necessarily the best tool for data analysis.

Product Segmentation Query Example

Lets look at an example ms access query or two and see how we can do the same thing Datamartist, only without the queries and without any SQL. For this example, lets say that we have two sets of sales data from different time periods, and a product list, and we want to define some product segments based on color and price. We want to get a summary of the sales Qty and average price sold by month, broken out by the new categories which are as follows:

• "Red and High Priced" If the product is Red and its minimum price is more than $1000
• "Red Low Price wide price range" If the product is Red, has a minimum price less than $1000 but has a min to max price of more than $200
• "Red Low Price small price range" If its Red and not in the first two segments
• "Yellow" if the product is yellow.
• "Other" for all the rest

The three data tables we have are as follows:

1. Sales 03-06 with about 120 000 rows, which contains sales data from 2003 - 2006
2. Sales 2007 with about 30 000 rows, which contains sales data for 2007
3. Products which contains the colors for all the products and their minimum and maximum prices

So- first step is to combine the two data tables, in Access, this is done with a UNION query with the following SQL code:

select * from [Sales Data 03-06] UNION select * from [Sales Data 2007];

In Datamartist, we simply connect the two tables up to a combine block.

Next, we need to define the segmentation- again in Access this is done with a Query, this time by nesting IIF statements to add a new column called "Product_Segment" to the resulting query.

SELECT Products.Product_ID, Products.Product_Name, Products.Product_Group, Products.Product_Category, Products.Product_SubCategory, Products.Shipping_Weight, Products.Color, Products.Price_Min, Products.Price_Max, IIf([Color]="Red" And [Price_Min]>1000,"Red and High Priced",IIf([Color]="Red" And ([Price_max]-[Price_min])>200,"Red Low Price wide price range",IIf([Color]="Red","Red Low Price small price range",IIf([Color]="Yellow","Yellow","Other")))) AS Product_Segment
FROM Products;

In Datamartist, we use a segmentation block to do the same thing. The interface is graphical, and the syntax is the same as you would use in Excel. There is no need to nest any IF statements, because the overall block is designed to do that. Heres what the blocks look like- the MS Access import block on the left, and the segmentation rule block on the right.

Each segment has the statement that defines if a row is in the segment or not. The block tests each segment rule in order, starting at the top- the first statement that solves as "TRUE" defines the value for the Product_Segment column for that row. Dragging the segments up and down changes what order the rules are checked.

(Click to Enlarge)

Then we have to Join this new product dimension (with the segmentation column) to the sales data, and summarize.

In MS Access, this is done with more queries- Heres what Access looks like when we're done.

Compare that list of Tables and Queries to the visual, left to right layout of the Datamartist data canvas that does the same thing. Without ever having to write any SQL code:

The VISUAL way to do it

In Datamartist you can see the flow of the data, the row counts are clearly displayed, and clicking on the connectors will bring up the underlying data set in the data viewer. Its clear which block feeds which, and by adding more blocks and connecting them at the desired point in the data flow, new analysis can be created.

Take Datamartist for a trial run- download it now because maybe you don't have to learn microsoft access queries after all.

• Item_Master.txt - this holds all the items that Acme buys
• Vendor_Master.txt - this holds a list of all the vendors, with information such as their address
• PO_Detail.txt - this is the huge data set, all the purchase order data for the last four years

The Item and Vendor files aren't very big, but the PO_Detail is over 340 Mb, and it holds almost four million purchase order lines. Don't try to import it into Excel. Of course you need Excel 2007 to even try to import 4 million rows. In Excel 2003 it would take over sixty sheets and probably some VBA code to try it. I tried the import in Excel 2007- it takes 20 seconds just to tell me I'll have to go back to the text file import multiple times to do multiple imports onto separate sheets. It took almost two minutes to do the first million rows. Even once we have the data spread across four sheets it's not clear how to summarize millions of rows in excel easily.

Instead, let's use the Datamartist tool to manage this data set and generate one thats more useful.

The first analysis we will do will be on the Vendor dimension, to determine who Acme's big vendors are, and if we can negotiate some price reductions where we have leverage.

In Datamartist, very large files are not an issue because the tool can load in only preview data- this means that it's possible to look at a sampling of a few hundred thousand rows, and design the transformation before running it on the whole data set.

The PO Detail file has the columns shown- let's answer the question - "Who are our biggest suppliers"?
So which columns do we need? We probably want to have some sense of trends over time so we'll keep the order date, but summarize to Month, we'll keep the Vendor ID of course, and then we need to use the Quantity and Price fields to calculate the total amount spent. Then we want to write this summarized data into Excel to check it out.

To do this in Datamartist all it takes is four simple blocks; A Text import block to load in the PO_Detail.txt file, a calculate block to multiply QTY by PRICE, a Summarize block to do all the summarizing, and an Excel export block to generate the excel file;

Each block passes its result to the next block via the connectors, and the last block saves it to an excel file we've specified.

Defining the calculation uses standard spreadsheet functions- here's what the config area looks like;

And defining the summary is as simple as it looks- pick the columns you want, and select what kind of summary you want done.

We run it on a preview set of 100 thousand rows (takes about twelve seconds to run), and check the output.

It looks good, so we run on the whole 4 million rows;

About seven minutes later we have our result- an excel sheet with a manageable 130 thousand rows, total spend, by vendor, by month for four years;

Next up we need to create our vendor dimension, and join it to this mini fact table we have created. Stay tuned.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

An Extract, Transform, Load (ETL) tool is an intermediate software application that extracts the data from the source system, transforms it (often another way to say it FIXES it) and then loads it into the destination system. They are also very expensive.

The destination system is usually a data warehouse or data mart, and most of the ETL tools available are server based.  The ETL tool and related development is key to any any data warehouse project (and represent a third or more of the cost on a typical project).

Although most ETL tools use a visual interface of one sort or another, at the core they require programming skills and specialized knowledge. Google "datastage training" and you'll see that there is an industry grown up around learning how to use these tools.

But there's nothing magical about it. If you ever made a spreadsheet with data from multiple sources, transformed the data, and then either made reports or moved the data into another spreadsheet then you have made (or most likely were an integral human part of) an ETL. The problem is that out of the box tools like Excel and Access are so flexible, that too much is possible.  Where to start?

The amazing thing is that EVERYONE needs ETL functionality, yet overwhelmingly the tools available are expensive, hard to learn and designed for the really, really heavy lifting.

Surely not every data manipulation task that is too much for Excel needs an enterprise ready server based ETL tool? Particularly in the current economic environment, oversized solutions are not an option.

A hard working analyst that has a bit of data analysis to do, and nothing but Excel or maybe Access on his/her desktop is short on options and long on messy spreadsheets or the need to "learn SQL in 21 easy steps".

The vision behind Datamartist is to provide an easy to use, powerful, yet low cost data transformation tool, that guides users to generate well structured data analysis sets.  And all at a price that represents less than a single day of those consultants you have to hire to use the other software you paid too much for.

This is the perfect time find out what easy, flexible, visual data transformation can be like-  download now.