This version of Datamartist brings with it some useful new functionality, including new functions that can be used in expressions, new capabilities in terms of exporting to databases, and a new data block.

In this post, we'll look at two new features, the Pivot block, and the enhanced database export capabilities.

Pivot Block

Our beta testers loved this new block. The pivot block lets you do the equivalent of a cross-tab query, rolling up a measure, and distributing the value in a new set of columns, where the column names are provided by the input data set.

Here is a simple example showing how it works:

Say we start with a set of data that has mutiple rows for each date, and different values in the color field, and a quantity measure:

Then we can connect one of the new pivot blocks to this data set like so:

The pivot block lets us select which columns to include (this defines the level of detail to roll up to), which string column to use to generate the new column names, and which measure to use as well as the rollup method (sum, average, min, max)

The result? The output of the pivot block looks like this: now we have a summary by color for each date, with a column for each color value.

Database export enhancements

Now, when exporting to a database, there are a number of new options.

One of the most interesting is the capability to execute SQL commands in the database either before and/or after the data is exported into the table.

This provides the capability of running stored procedures, or launching follow on database side processing after Datamartist writes the data into the DB.

This is a powerful new capability, and makes it even easier to integrate datamartist into various systems, and get your data quality and profiling data where you need it.

If you haven't checked out datamartist yet, we're not sure what you are waiting for- download the free trial, and give it a go.

We've had lots of great feedback from our customers and are thrilled with how people are using Datamartist, not just for powerful and flexible data profiling, but for data migration, data quality work and ad-hoc datamart creation.

As always we're committed to continually improve our products- Here are just a few of the features added in this latest version;

Block definition import and export

Datamartist adds a new level of reuse-ability and collaboration capabilities with the addition of block export/import.

This lets you export a block configuration (or a number of blocks, with all their connectors) to a file that can then be imported into any other Canvas, either by yourself or by your colleagues that are also using Datamartist. Just select the blocks you want, and right click to export- just right click anywhere on any canvas to import an existing block file.

We've found this particularly useful in saving filters, segmentations, and even full data profiling blocks to give us a library of useful blocks and block groups that we use again and again.

Improved database connectivity and connection management

We've also added some features and improved how we connect to databases in Datamartist

Datamartist can connect to SQL Server, Oracle, MySQL, MS Access, Text files and Excel files, as well as having and ODBC driver that lets you connect to many other databases. Now its even easier to manage a large number of database connections for multiple database types- keep all those servers and connections at your fingertips when you are combining all that data!

Data profiling in an affordable, graphical, visual environment

Find out why people are loving the combination of an ETL and a Data profiling tool in one- using Datamartist not just for data profiling, but for data migration, data quality audits, and ad hoc datamart creation. Try the free trial today.

The trick with public data like this is that often (ok almost ALWAYS) either data is missing, or the codes don't match up. In this case, the country descriptions from various data sets I want to use don't match- and different data sets have different holes (i.e. not all datasets include data for all countries). Finally, some data sets have a different definition of a country- for example, they break out places like "British Indian Ocean Territories" that need to get rolled up in the UK numbers.

Country statistics data such as GDP, GNI and Population were also incorporated to provide dimensions to carry the analysis out on. The raw trade data was obtained from Industry Canada's "Trade Data Online" (http://www.ic.gc.ca/sc_mrkti/tdst/tdo/tdo.php?lang=30&productType=HS6). The World Bank was the source of the country statistics data (http://data.worldbank.org/indicator). A zip file containing the raw data, as well as the Datamartist data transformation .dmc file is provided at the bottom of this post.

I started by transforming the country statistics data. The raw data included information on GDP, GNI, Total Population and Urban Population. A screenshot of the Datamartist canvas for the first portion of this data transformation is provided below.As seen above, the first step in the data transformation involved importing the four excel data files. During this import, columns with data not relevant to the year 2009 were filtered out and zeros were inserted into any null data rows, signifying that data for that row was not available. A series of data "Join" functions were then carried out to create one data file containing all of the country statistics information. Upon completion of joining these data files," a "Calculation" block was utilized to replace any null data values resulting from the data join with zero's. Finally, the country statistics information was joined with a country cross reference list. Basically, this join standardizes all of the country names.

The second part of the country statistics data transformation focused on segmenting the data, as shown in the Datamartist canvas screenshot below.Before the data could be segmented, it was summarized so that there was only one data row for each standardized country name. A "Calculation" block was then added to calculate the GDP per Capita, GNI per Capita and Urban Population Percentage using the Population data. With these calculations complete, a series of "Segment" blocks were added to the canvas. The "Segment" blocks are extremely useful because they add an additional column to the data set which is populated according to a set of segmentation rules defined by the user. In this example, the "Segment" block was used to segment the GDP per Capita, GNI per Capita, Urban Population Percentage and Population data. The segmentation rules for the Population "Segment" block are shown in the screenshot on the left.

A similar set of data transformations was also carried out on the Canadian Import and Export Trade data. As seen in the Datamartist canvas screenshot, the Canadian Import and Export Trade data was imported, joined and null data values were replaced with zeros. The country names were then standardized and summarized so that the Canadian Import and Export Trade data could be joined with the Country Statistics data.

With all of the raw data transformed into a suitable format, a final set of data transformations were carried out to create a single text file. This text file was then exported so that QlikView could be used to analyze the data. A screenshot of the Datamartist canvas for this final set of data transformations is shown below. In this final set of data transformations, the "Star Schema" block was used first. The "Star Schema" block is a handy data transformation tool because it allows numerous data join operations to be carried out simultaneously. It was used to combine the Country Statistics data and Canadian Import and Export Trade data with data defining a country's geographical region. A screenshot of the "Star Schema" block configuration window is shown to the left. The joined data was then put through a "Calculation" block one last time to eliminate any null data values. Finally, the transformed data was exported as a text file so that it could be analyzed in QlikView.

The transformed data was then imported into QlikView and a dashboard was created to analyze the data with. QlikView is a great data analysis tool because it allows data to be filtered and visualized very efficiently. In this example, I made a dashboard that allows Canada Import and Export Trade data to be visualized using the Country Statistics Data segments created using the Datamartist software as filters. A screenshot of the dashboard with no filters applied is shown to the right. I am now going to show a series of screenshots with different data filters applied. To start off, I want to see the countries that Canada Exports the most goods to. To do this, I just dragged a box over the largest bars on the export graph as seen below. Once I finished making the data selection box on the graph, I released the mouse and QlikView automatically zoomed in on the area I selected. In addition, the upper right table in the dashboard updates as well. The image below shows the results. As seen in the image, Canada's biggest export trade partners in 2009 were the United States, the United Kingdom and China. As a second example, I will filter the data using the data segments created in Datamartist. If I click on "Asia" in the "Region" box then only data from the countries in Asia is shown in the table and graphs. Furthermore, the segments in the other filter boxes (GDP per Capita, GNI per Capita, etc) updates to the region selection as well. It does this by highlighting the data filter segments that are valid for the "Asia" region in white. For example, in the "GDP per Capita" box, all data segments are valid except for the "GDP > $100 Thousand" segment. A screenshot of QlikView with the "Asia" region filter on is shown below. I can further filter the data by clicking on any other data filter segments that are white. As an example, if I select "$1 Thousand - $5 Thousand" in the "GDP per Capita" box and "40% - 60%" in the "Urban Population Percentage" box, the graphs and table update again. In this instance, the only countries that meet these filter requirements are China, Georgia and Mongolia. A QlikView screenshot with all three of the filters chosen is shown below.

Try it out yourself with the free trial

You can give Datamartist a try with this data, just signup and download the free trial, and then download a zip file will all the data, and the example Datamartist file.

Just extract all the files in the above ZIP file into the "My Datamartist" folder that the Datamartist trial will create when you run it, and open the "World Trading Example.DMC" file with Datamartist.

You'll find that Datamartist gives you a powerful, visual way to transform data from lots of places, and get it ready for great visualization tools like Qlikview in a step by step, clean, repeatable way.

On top of that, datamartist can be automated- so if you have data transformations you need to run on a schedule, you can design them in a graphical environment, test them, and then have them run automatically.

Once again, the U.S. Energy Information Administration was the source of the data (http://www.eia.doe.gov/cneaf/electricity/esr/esr_sum.html, Table 10). This time, the data was provided as one Microsoft Excel spreadsheet giving detailed information about the thousands of electric power generation companies in the U.S.A. The raw Excel file from the website had some merged cells, and had column names in multiple rows, so we had to clean it up a bit before importing. This is often the case, when using files that are more reports than raw data.

Even after these formating fixes, as is often the case, the data in its original form is not what we need- due to the large size and slightly different formatting of this spreadsheet, it was necessary to transform the data before combining it with the electricity consumer data. A visual map of these data transformations is provided in the screen shot of the Datamartist canvas below.

As seen in the screen shot above, the first data transformation carried out was a data summarizing operation. In this operation, the input data was summarized by state and class of ownership, converting the 3200 row input data file into a data file containing only 170 rows of data.

The next data transformation step required the use of the "Join" function. This is a very useful function in the Datamartist software because it allows two data files with different formatting to be joined together into one data file. In this instance, the summarized electricity generation data file was joined with a data file containing U.S.A. state abbreviations. This was done because the U.S.A. electricity generation input data file only contained state abbreviations. The result of this joining operation inserted a column containing the states full name based on the state abbreviation used in input data file.

With the join operation complete, the data was further transformed using the "Calculate" and "Sort" functions. The "Calculate" function was used to remove the data column containing the state abbreviations, as the previous "Join" function had added a column with the states full name. The "Sort" function was then used to sort the electricity data by state and class of ownership.

The data was then put through one last calculation function that renamed the column names, before being exported as a text file.
Now that the electricity generation data has been transformed, the Tableau visualizations can be used to present the data. It is worthwhile noting that since I joined the electricity consumption data with the electricity generation data before exporting it from Datamartist, I only needed to import one data file into Tableau Public 5.2. This single data file contained all of the information required to create both the electricity consumption and generation visualizations. As with part one of this blog post, I have included a Tableau dashboard summarizing state level electricity generation statistics below. This visualization is very similar to the visualization in the first part of this blog post. You can try it for yourself by clicking a state on the map and watching the table below the map update to present data about that particular state.

See the Data Transform Yourself!!!
All of the data files used in the data transformations in part one and part two of this blog post, as well as the Datamartist .dmc file, can be downloaded in a ZIP file here. You can see the data transformations discussed in these blog posts for yourself by downloading the free trial, install Datamartist, and then put all the files in the above zip file to the "My Datamartist" folder that datamartist creates in your "Documents" folder. Then just open the "Electricity Example.dmc" file in Datamartist and check it out.

Rather than looking at the actual individual values for a given column, by profiling the structure of the values you can understand at a higher level the quality of data. This technique is primarily used for string based data.

Can't find the forest because of the trees?

Lets look at a simple but very real example. You have a data set with a few tens of thousands of rows that includes a text field for telephone number.

What kind of phone numbers do you have?

Nobody wants to look through 40,000 rows value by value to see where the bad records are.

Profiling with character substitution and elimination

With a data profiler you can define rules:

1. IF the character is a digit (0-9) then replace it with the letter "n"
2. IF the character is a letter (a-z) then replace it with the letter "a"



These two simple rules turn 40,000 rows of phone numbers into a short list of different phone number formats- and we can see which ones seem valid or seem to have problems.

By adding one more rule, eliminating a few characters that do not directly affect the phone number, we can reduce the list even further.


The last rule ignores the space, open and close brackets and dash characters- and that simplifies the different formats in the analysis to a mere five formats;

1. 10 digits
2. 11 digits
3. 9 digits
4. A 7 character string (drilling down by clicking on this bar reveals the word "unknown"- not so useful.)
5. Missing.



Of course, data format profiling does not always give a final answer regarding data quality. In this case, just because a phone number has the right number of digits, does not mean that it is a valid phone number, and even if its a valid phone number it may not be the right phone number for that customer...

Data format analysis for structured codes

This same technique is also useful for any fields that are meant to contain structured codes.

For example, say a valid product code is supposed to start with one letter, where the first letter should be one of A,B,D or G be followed by a dash, and then four digits.

So A-2324 is a valid code, but M-2334 and J234 are not, for example.

The following rules would help detect a validly formatted string (and point out the issues:)



The result on a mocked up set of data:



Using data format patterns to examine the contents of a string column is a very useful way to start to understand what's in the column. More than just a Yes-No result, it actually gives you a visual look at what types of issues exist.

The analysis here was done using the Datamartist tool, an easy to use data profiler and data transformation tool. To try making some data format rules for your own data, give the free trial a try.

Up next in our data profiling series of blog posts- an even more powerful (although often more complex) technique called "Regular Expressions" or "Regex". This specification language can define complex rules that analyze strings and determine if they belong to a given set (say, "Valid product codes" as in this example) or not.

]]> http://www.datamartist.com/data-profiling-rules-and-data-formats/feed 0 http://www.datamartist.com/data-profiling-enhanced-datamartist-v1-3-released http://www.datamartist.com/data-profiling-enhanced-datamartist-v1-3-released#comments Tue, 21 Sep 2010 18:55:08 +0000 James Standen http://www.datamartist.com/?p=5050 We are very pleased to announce that Datamartist V1.3.0 is now available, and want to thank all our Beta testers. This release represents a major step forward particularly in the data profiling tools area for the professional edition. Datamartist gives you a highly visual, drag and drop, drill down, free form data profiling and transformation tool for much less than enterprise tools from "those other guys". For a limited time, you can get the new version of Datamartist at the old price. But After Sept. 30, the discount is gone.

Check out the two minute whirlwind tour video.

What's new in V1.3

Just a few of the great new things you'll find in this version, along with all the usual visual, drill downable data profiling and transforming goodness;

Value distribution data profiling

Quickly analyse the distribution of numeric values zooming in on questionable distribution areas, exploring visually, going from visual to drilling down into the underlying rows and back instantly. You can check it out in this video.

Automated data profiling

We've added a dedicated data profiling block to the block library of the professional edition that lets you sample as many points of the data canvas as you want, and write time-stamped data profiling results. The result is that not only can you profile your data automatically, you can analyse trends, and use the data profiling stream from Datamartist to create external dashboards and data quality alerts.

Enhanced data format rule editing

The professional edition also boasts an enhanced ability to directly edit data format analysis rules. By defining a series of simple rules, you can analyse the data format of your data quickly and easily. Are there letters where you should only have numbers? Are the product codes properly formed?

Regular Expression support (REGEX)

Datamartist professional now has a REGEX function that allows you to include regular expressions anywhere that expressions are used. This means you can use them in calculation blocks for transforms, segmentation tests, and filters. This gives you a powerful new tool to assess and profile your data, either writing your own, or using one of the many expressions available.

For a more detailed look at Datamartist, you can browse the documentation, or better yet, download the free trial now, and see what Datamartist can do with your data.

]]> http://www.datamartist.com/data-profiling-enhanced-datamartist-v1-3-released/feed 0 http://www.datamartist.com/datamartist-v1-3-0-value-distribution-data-profiling http://www.datamartist.com/datamartist-v1-3-0-value-distribution-data-profiling#comments Mon, 26 Jul 2010 18:33:50 +0000 James Standen http://www.datamartist.com/?p=4855 This video gives a quick (under two minute) look at the Datamartist data profiler's ability to explore the distribution of numeric values in a data set by counting the number of values that fall into a series of equal size buckets. It highlights the datamartists calculation, visualization, selection and drill down features using a simple example.

This value profiling tool is just one of many of the Datamartist data profiling tools capabilities, download the free trial of the BETA to try all the functionality with your own data.

]]> http://www.datamartist.com/datamartist-v1-3-0-value-distribution-data-profiling/feed 0 http://www.datamartist.com/v1-3-0-public-beta-data-profiling-tools-enhance http://www.datamartist.com/v1-3-0-public-beta-data-profiling-tools-enhance#comments Wed, 14 Jul 2010 02:45:17 +0000 James Standen http://www.datamartist.com/?p=4676 Come and get it while its still warm! The next release of Datamartist, a data profiling and data transformation tool (think ETL and data profiler rolled into one) is now available in BETA as a public trial download. UPDATE: V1.3 has been released- thanks to all our Beta testers!

The currently released version 1.2.6 is of course also still available, but for those who don't mind risking a bug or two (we need your help in killing the last few) the Beta gives you a sneak peak at the new version and a bunch of new features that have been introduced to the Professional edition.

Whats in it? Lots of data profiling goodies;

Value distribution profiling

One of the important additions to the data profiling capabilities in Datamartist Pro is the value distribution explorer. This powerful functionality analyzes numeric fields and provides a value distribution graph (equal size buckets for row counts based on value) that lets you zoom in and out, and drill down into the rows to understand your data and spot any suspicious values at a glance.



Support for Regular expressions for pattern matching

The new version has added a very powerful function to the function library in the professional edition;

REGEX(text,regex expression)

With this function, it is now possible to use regular expressions (regex expressions) to evaluate if string values conform to desired data formats. Regular expressions are widely used for data quality testing, and there are lots of them available out there.

Custom data format rules

In V1.3.0 professional edition, it is now possible to add a series of custom rules for character mapping, creating much more flexibility in the data profiling tool. For example, its possible to map numbers, letters, punctuation characters, or any combination of them.

Data profiling block

In the professional edition, the category of data quality blocks has been added, and the first block released is the data profiler block.

This lets data profiling results themselves be used within your data canvas, as well as written out to files or database tables, with time stamps- enabling data profiling automation and tracking. It is now possible to define, calculate and track data profiling metrics automatically, sampling data at a sample frequency that lets you understand how your data quality is evolving.

We're excited about what can be done with this first data quality block- and we are sure our professional edition users will be tracking their data quality like never before.



I'll be blogging about all the new features in the coming days and weeks- I'm certain you'll find lots of things to like in the new version- give it a go.

As always with our betas, we love feedback, and we'll be giving some free PRO licenses away to our most active Beta testers.

]]> http://www.datamartist.com/v1-3-0-public-beta-data-profiling-tools-enhance/feed 0 http://www.datamartist.com/data-profiling-do-it-do-it-now http://www.datamartist.com/data-profiling-do-it-do-it-now#comments Fri, 07 May 2010 02:11:58 +0000 James Standen http://www.datamartist.com/?p=4496 Imagine that you have bought a new home, and you've decided to do some landscaping. So you pick three landscapers, draw a rough sketch of what you want, and ask them to bid on the job.

But you don`t allow them to come see your property, and your sketch doesn't specify anything about the existing landscaping- just the final configuration. Do you think the landscapers would be willing to offer a reasonable price ?

Unlikely. What if there are existing patio stones to remove- or an in-ground swimming pool that`s got to go?

No landscaper would take on a job without understanding the lay of the land, and the existing conditions. It would be impossible to estimate the job. Anyone who did would give you a huge price to cover themselves, or demand extras upon discovering the extra work.

Yet when companies hire consultants to build them business intelligence solutions, or do data migration, it often happens with only the roughest outline of the existing data sets. Certainly, often a data model is included- but knowing what the table SHOULD contain rather than what it does is just not the same thing. It never ceases to amaze me that the simple, cost effective practice of data profiling is just often not part of the initial phases of so many business intelligence and data migration projects.

With the right data profiling tool, and just a few days work, its possible to gain a huge amount of insight into the data quality in your systems, and as a result, be able to make radically more accurate estimates of the cost to go from the "as is" to the "to be".

Phil Simon talked about this in a great post on the Data flux blog called "What Consultants Don't tell you", and raises an important and somewhat ugly truth- many times, service providers don't WANT to do data profiling because it reveals the true extent of the work to be done, increasing the budget requirement, and makes the project less likely to be approved.

Now certainly, we can't use a broad brush to paint all consultants, but it does lead to a reduction in the number of times valuable tools such as data profiling are recommended even though in my opinion they are a low cost, no-brainer, do it unless you are crazy first step to any major project.

You are going to spend potentially millions of dollars on a business intelligence or data migration project- spend a few weeks to look at the data with the right tools first for goodness sake!

If you want to get a reasonable cost estimate, and you want to go into your business intelligence or data migration project with open eyes, don't imagine you can know what it will cost to get from here to there if you don't take a good look at where here really is.

Full disclosure- of course, you are reading the Datamartist blog, and Datamartist has lots of data profiling functionality- so you have to understand that we are incredibly biased on this topic. If you are able to overlook our inherent bias, give the tool a try- you`ll discover things about your data you might not have wanted to know, but its better to face the truth prepared, than to rely on wishful thinking, and then discover the bad news when you're well into the project, and your budget is almost gone.

]]> http://www.datamartist.com/data-profiling-do-it-do-it-now/feed 2 http://www.datamartist.com/datamartist-v1-2-now-available http://www.datamartist.com/datamartist-v1-2-now-available#comments Tue, 02 Mar 2010 14:45:33 +0000 James Standen http://www.datamartist.com/?p=4266 nModal solutions is pleased to announce that Datamartist V1.2 is now available.

In this version, we've introduced a Standard and Pro edition, letting customers get the features they need at the right price.

• Datamartist Standard: $349
• Datamartist Professional: $745

A comparison of the feature sets explains the details.

Whats new in V1.2

Data source import enhancements

• Ability to cut and paste between Excel, Text files, the Datamartist canvas and any Datamartist data viewer.
• New integrated data source repository with drag and drop to canvas.
• SQL Editor to allow the creation of SQL queries to get data from databases.



Running Datamartist canvases automatically

Now that Datamartist can be run from the command line, it is possible to schedule datamartist transforms- even running it on a Windows server. Details about the logging and options are here.


Edit Internal data sets.

The addition of fully editable internal data sets that are stored within the DMC file itself gives a powerful new ability to create "What if" type scenarios. Imagine you want to see the effect of changing the sales regions slightly- just copy and paste the existing from a data viewer onto the canvas- that gives you an internal data set block with that data in it- now you can add a column "New Region" or rename the column, then edit some values, join it back into the original data with a join block, and be trying different scenarios in no time.


We're excited about this new release, and thanks to all our customers and testers for their feedback- we're glad to be incorporating some of those great ideas into the product.

If you haven't tried Datamartist yet, this is the perfect time, and now with two editions to choose from you can get the features you need at the right price.

]]> http://www.datamartist.com/datamartist-v1-2-now-available/feed 0