This value profiling tool is just one of many of the Datamartist data profiling tools capabilities, download the free trial of the BETA to try all the functionality with your own data.

The currently released version 1.2.6 is of course also still available, but for those who don't mind risking a bug or two (we need your help in killing the last few) the Beta gives you a sneak peak at the new version and a bunch of new features.

You can get it here. Whats in it? Lots of data profiling goodies;

Value distribution profiling

One of the important additions to the data profiling capabilities is the value distribution explorer. This powerful functionality analyzes numeric fields and provides a value distribution graph (equal size buckets for row counts based on value) that lets you zoom in and out, and drill down into the rows to understand your data and spot any suspicious values at a glance.

Support for Regular expressions for pattern matching

The new version has added a very powerful function to the function library;

REGEX(text,regex expression)

With this function, it is now possible to use regular expressions (regex expressions) to evaluate if string values conform to desired data formats. Regular expressions are widely used for data quality testing, and there are lots of them available out there.

Custom data format rules

In V1.3.0, it is now possible to add a series of custom rules for character mapping, creating much more flexibility in the data profiling tool. For example, its possible to map numbers, letters, punctuation characters, or any combination of them.

Data profiling block (Pro edition)

In the professional edition, the category of data quality blocks has been added, and the first block released is the data profiler block.

This lets data profiling results themselves be used within your data canvas, as well as written out to files or database tables, with time stamps- enabling data profiling automation and tracking. It is now possible to define, calculate and track data profiling metrics automatically, sampling data at a sample frequency that lets you understand how your data quality is evolving.

We're excited about what can be done with this first data quality block- and we are sure our professional edition users will be tracking their data quality like never before.

I'll be blogging about all the new features in the coming days and weeks- I'm certain you'll find lots of things to like in the new version- give it a go.

As always with our betas, we love feedback, and we'll be giving some free PRO licenses away to our most active Beta testers.

But you don`t allow them to come see your property, and your sketch doesn't specify anything about the existing landscaping- just the final configuration. Do you think the landscapers would be willing to offer a reasonable price ?

Unlikely. What if there are existing patio stones to remove- or an in-ground swimming pool that`s got to go?

No landscaper would take on a job without understanding the lay of the land, and the existing conditions. It would be impossible to estimate the job. Anyone who did would give you a huge price to cover themselves, or demand extras upon discovering the extra work.

Yet when companies hire consultants to build them business intelligence solutions, or do data migration, it often happens with only the roughest outline of the existing data sets. Certainly, often a data model is included- but knowing what the table SHOULD contain rather than what it does is just not the same thing. It never ceases to amaze me that the simple, cost effective practice of data profiling is just often not part of the initial phases of so many business intelligence and data migration projects.

With the right data profiling tool, and just a few days work, its possible to gain a huge amount of insight into the data quality in your systems, and as a result, be able to make radically more accurate estimates of the cost to go from the "as is" to the "to be".

Phil Simon talked about this in a great post on the Data flux blog called "What Consultants Don't tell you", and raises an important and somewhat ugly truth- many times, service providers don't WANT to do data profiling because it reveals the true extent of the work to be done, increasing the budget requirement, and makes the project less likely to be approved.

Now certainly, we can't use a broad brush to paint all consultants, but it does lead to a reduction in the number of times valuable tools such as data profiling are recommended even though in my opinion they are a low cost, no-brainer, do it unless you are crazy first step to any major project.

You are going to spend potentially millions of dollars on a business intelligence or data migration project- spend a few weeks to look at the data with the right tools first for goodness sake!

If you want to get a reasonable cost estimate, and you want to go into your business intelligence or data migration project with open eyes, don't imagine you can know what it will cost to get from here to there if you don't take a good look at where here really is.

Full disclosure- of course, you are reading the Datamartist blog, and Datamartist has lots of data profiling functionality- so you have to understand that we are incredibly biased on this topic. If you are able to overlook our inherent bias, give the tool a try- you`ll discover things about your data you might not have wanted to know, but its better to face the truth prepared, than to rely on wishful thinking, and then discover the bad news when you're well into the project, and your budget is almost gone.

In this version, we've introduced a Standard and Pro edition, letting customers get the features they need at the right price.

• Datamartist Standard: $349
• Datamartist Professional: $745

A comparison of the feature sets explains the details.

Whats new in V1.2

Data source import enhancements

• Ability to cut and paste between Excel, Text files, the Datamartist canvas and any Datamartist data viewer.
• New integrated data source repository with drag and drop to canvas.
• SQL Editor to allow the creation of SQL queries to get data from databases.

Running Datamartist canvases automatically

Now that Datamartist can be run from the command line, it is possible to schedule datamartist transforms- even running it on a Windows server. Details about the logging and options are here.

Edit Internal data sets.

The addition of fully editable internal data sets that are stored within the DMC file itself gives a powerful new ability to create "What if" type scenarios. Imagine you want to see the effect of changing the sales regions slightly- just copy and paste the existing from a data viewer onto the canvas- that gives you an internal data set block with that data in it- now you can add a column "New Region" or rename the column, then edit some values, join it back into the original data with a join block, and be trying different scenarios in no time.

We're excited about this new release, and thanks to all our customers and testers for their feedback- we're glad to be incorporating some of those great ideas into the product.

If you haven't tried Datamartist yet, this is the perfect time, and now with two editions to choose from you can get the features you need at the right price.

Tableau has received a fair amount of positive response from the visualization world. Even Stephen Few, who isn't shy to point out when visualizations are straying from the straight and narrow has been supportive of Tableau from the start.

We're excited about this new access to such a great data visualization tool because we know that people who do visualization have to transform their data- and if users of Tableau want a flexible, visual data transformation tool, the Datamartist tool is an obvious choice.

Free Tableau vs Professional version

There are of course some limitations with the free version of Tableau, in comparison with the full featured professional version ($1600 USD per seat.)

• Data import capabilities - Only MS Access, Excel and Text files
• 100 000 row Limit per table
• 50 Mb Limit per organization for the web server.

None of these limitations will stop you from making some pretty fantastic visualizations as long as your final summarized data set fits within the limits and you put it in the right format. But it does mean that to use this version of Tableau, you need to use another tool to get those large data sets summarized. This is probably something you are doing anyway, because there is almost always some data cleanup to do.

Get your data sets ready- there are going to be some beautiful viz getting made

The trick with visualization of course, is that you need data to visualize. With tools like Tableau, as good as they are at making the pictures, you have to get the data set to them first- and we all know how many pre-formatted, all is well, no data quality issues data sets there are lying around the real world (Hint: none.).

But I'll tell you, once you do have the data, what a fantastic bit of interactive web based visualization Tableau can do.

Often, people use a combination of MS Excel and Access to create the datasets that they want, then connect to the Access database with Tableau. Of course, we suggest you try Datamartist instead. Datamartist and Tableau are a powerful combination- first, using Datamartist pull data from mutliple sources (Datamartist loads out of SQL Server, Oracle, MySQL, and at a license price much less than Tableau Professional).

Datamartist lets you join tables visually (using a venn diagram interface that we're proud of), segment data using rule sets, summarize millions of rows if needed, and generally parse and transform with an easy to use calculation engine. Once you have the data where you want it, export it easily to an Access database, and let Tableau Public generate the visualizations that you need.

I'm excited about Tableau's decision to make its power available in this public version. I intend to do some serious data crunching with the Datamartist Tool, followed by some interactive visualizations with Tableau Public. The beauty of the Tableau Public setup is that I can then publish the visualizations right here in the blog, and highlight what the combination of these two tools can do.

Joins can be said to be INNER or OUTER joins, and the two tables involved are referred to as LEFT and RIGHT. By combining these two concepts you get all the various types of joins in join land: Inner, left outer, right outer, and the full outer join.

Tables used for SQL Examples

In the screen shots I've configured Datamartist to only show the name columns to save space. The SQL code shown is "Select *" so it will return all the columns. You can see that in the Datamartist tool the type of join is selected by just checking the parts of the venn diagram that contain the rows you want.

1) Inner Join SQL Example

select * from dbo.Students S INNER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

2) Left Outer Join SQL Example

select * from dbo.Students S LEFT OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

4) Full Outer Join SQL Example

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

5) SQL example for just getting the rows that don't join

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID where A.Advisor_ID is null or S.Student_ID is null

6) SQL example for just rows from one table that don't join

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID where A.Advisor_ID is null

But what about the duplicate row thing?

Now, since in this case we had a simple one to one relationship, the number of rows that were returned made the venn diagrams make sense, and add up pretty normally with table one and two.

What happens if the data in the tables are not a simple one to one relationship? What happens if we add one duplicate advisor with the same ID, but a different name?

A join will create a row for every combination of rows that join together. So if there are two advisors with the same key, for every student record that has that key, you will have two rows in the inner part of the join. The advisor duplicate makes duplicate student records for every student with that advisor.

You can see how this could add up to a lot of extra rows. The number of rows is the product of the two sets of joining rows. If the tables get big, just a few duplicates will cause the results of a join to be much larger than the total number of rows in the input tables- this is something you have to watch very carefully when joining- check your row counts.

So there you have it. If you want to try joining tables with the Datamartist tool- give it a try. It's a super fast install, and you'll be joining like a pro in no time.

Thanks to Geoffrey Wiseman who covered the event on his blog Tech scene, you can watch the video of the presentation.

It was lots of fun, and we got some great feedback. A few people came up after and used the term "Excel Hell" to describe some of their previous data transformation experiences, and they were right into what Datamartist could do.

Note: The event was held in an auditorium in one of the Rogers offices (which explains the Rogers logo on the podium in the photos)- Datamartist has no affiliation with Rogers of course, but we're grateful for all the support they give the Toronto startup community through sponsoring Democamp and providing such a great space for the sold out event.

Sneak preview

We've still got a heavy development cycle in December, but we're excited about the release we're targeting to get out the door early in the new year. Because we worked to put together a specific build for this demo, and it's pretty stable, I've made the decision to make this beta publicly available- if you go to the download page you'll find a link at the bottom that will let you download the installer. As with anything beta, it might still have some bugs in there- but overall you'll find some key new functionality that went over very well here in Toronto:

• Drag and drop for file import- just drag flat files or excel files onto the Datamartist canvas, and the blocks are instantly generated, ready to go.
• Copy and paste data - Copying a range in excel, then right clicking or pressing CTRL-V in Datamartist generates a new kind of block called a "Internal data set". More about these blocks soon, as we have lots of plans for them in this upcoming release.
• Copy and paste subsets from data viewer- Now you can also copy and paste data internally- click on a stub of a block, select only the rows, or only the columns, or even the cells that you want, press CTRL-C then right click on the canvas and you can paste that data into a new internal data set block.

Of course, copy and paste has some limits in terms of data sizes, but it is very useful for reference sets and can still handle pretty good sized sets.

The whole drive in continuing to enhance this type of functionality is about making an environment where its easy to move data around, in and out, and transform it- being able to explore the data and get it right where you want it in an intuitive, flexible environment. Stay tuned, lots more to come!

This version is fully backward compatible with all previous versions of Datamartist, and we encourage all our customers to upgrade.

This version of Datamartist brings with it a number of enhancements, some of the highlights:

Native data base import/export for Oracle and MySQL Enterprise Server

We're very happy to have added two more very significant database vendors to the natively supported list, and expect more in the future. In addition to adding the connectivity, we've also made enhancements to the general import interface, particularly in the selection and browsing of tables from large databases. Check it out.

A new Join block that lets you join inner, outer, right and left to your hearts content;

We are particularly proud of this block, because it gives a clean, clear visual view into one of the most important data analysis functions, the table join. You can specify the scope of the join easily using the venn diagram, and can drill down into the various result sets and see whats joining and whats not.

Enhanced copy and paste.

We've made almost all the data grids selectable and copy enabled, so you'll find that you can easily grab a value, a row, a columns, or any combination, copy it, and then paste it into Excel or other applications. We've been loving the flexibility this gives us, and hope you like it too.

We're always looking for feedback from our users, so let us know what you think.

Data profiling

Datamartist now has integrated data profiling capabilities that can be used at any point in a data canvas. By simply clicking on any block stub or data connector between blocks the data profiler is available at the bottom left of the screen:

The profiler provides a number of different metrics for each column:

• Column Name- the name of the column being profiled.
• Null rows- the count of rows that contain the null value.
• Missing rows- for string types the count of rows containing the empty string.
• Populated rows- the count of rows that are not Null or missing.
• Completeness- the percentage of rows that are Populated (Populated/Total Rows).
• Cardinality- the number of unique values contained in the row.
• Uniqueness- the cardinality divided by the number of populated rows.

The tree control visible on the left can be expanded and is linked- as you click on either a category value or the bars in the bar chart, you can drill down to the specific rows underlying that category, format or value type.

Field Formats

The data profiler also parses the values in each column using a format code as follows- this allows you to at a glance understand issues with poorly formed strings (such as Zip codes or Phone numbers)

• a- Denotes a letter of the alphabet
• n- Denotes a digit (0 to 9)
• s- Denotes white space (such as spaces)
• Punctuation is passed through to the format string so that you can see locations of brackets and dashes etc.

With this added functionality you can quickly find data quality issues and then use Datamartist's transformation functionality to resolve them. Once you've added and configured the blocks to fix it, you can verify that the format is good by simply running the data profiler on the resulting output.

Import directly from Excel spreadsheets

We've also added a direct connection to excel to allow data sets to be imported from one or more sheets or ranges within Microsoft Excel spreadsheets. While the data limitations in Excel make this more useful for smaller datasets, its a handy way to get reference information into your Datamartist canvases. We can probably all admit to having a bit of data tucked in an Excel sheet somewhere that we want to get at from time to time.

Export to database

Another very useful addition to the datamartist block library is the Export to DB block. With this block, it is now easy to write the results of your datamartist canvases directly to a new table in a database. Currently, access, SQL Server and a generic ODBC connector are supported- but watch for more native database connectivity soon.

We are determined to keep adding the functionality that our users need. We want to take this opportunity to thank our customers for their feedback, it makes it possible for us to keep improving our product, and keep to our vision of providing the right functionality in an easy to use desktop ETL tool. Our goal is to not have more functions than those expensive enterprise ETL tools, but to have the functions that our users need- and to package those functions in a simple but powerful interface. Give Datamartist a try, and tell us what you think.

We would very much like to thank all those who participated in the public beta, your testing and valuable suggestions are helping us deliver on our vision of an entirely new type of data transformation tool.

A very special and limited time introductory discount of 50% from the list price is being offered to encourage early adopters, with free upgrades for one year from purchase, to allow users to lock in the low price while enjoying the continued evolution of the product going forward.