This version of Datamartist brings with it some useful new functionality, including new functions that can be used in expressions, new capabilities in terms of exporting to databases, and a new data block.

In this post, we'll look at two new features, the Pivot block, and the enhanced database export capabilities.

Pivot Block

Our beta testers loved this new block. The pivot block lets you do the equivalent of a cross-tab query, rolling up a measure, and distributing the value in a new set of columns, where the column names are provided by the input data set.

Here is a simple example showing how it works:

Say we start with a set of data that has mutiple rows for each date, and different values in the color field, and a quantity measure:

Then we can connect one of the new pivot blocks to this data set like so:

The pivot block lets us select which columns to include (this defines the level of detail to roll up to), which string column to use to generate the new column names, and which measure to use as well as the rollup method (sum, average, min, max)

The result? The output of the pivot block looks like this: now we have a summary by color for each date, with a column for each color value.

Database export enhancements

Now, when exporting to a database, there are a number of new options.

One of the most interesting is the capability to execute SQL commands in the database either before and/or after the data is exported into the table.

This provides the capability of running stored procedures, or launching follow on database side processing after Datamartist writes the data into the DB.

This is a powerful new capability, and makes it even easier to integrate datamartist into various systems, and get your data quality and profiling data where you need it.

If you haven't checked out datamartist yet, we're not sure what you are waiting for- download the free trial, and give it a go.

We've had lots of great feedback from our customers and are thrilled with how people are using Datamartist, not just for powerful and flexible data profiling, but for data migration, data quality work and ad-hoc datamart creation.

As always we're committed to continually improve our products- Here are just a few of the features added in this latest version;

Block definition import and export

Datamartist adds a new level of reuse-ability and collaboration capabilities with the addition of block export/import.

This lets you export a block configuration (or a number of blocks, with all their connectors) to a file that can then be imported into any other Canvas, either by yourself or by your colleagues that are also using Datamartist. Just select the blocks you want, and right click to export- just right click anywhere on any canvas to import an existing block file.

We've found this particularly useful in saving filters, segmentations, and even full data profiling blocks to give us a library of useful blocks and block groups that we use again and again.

Improved database connectivity and connection management

We've also added some features and improved how we connect to databases in Datamartist

Datamartist can connect to SQL Server, Oracle, MySQL, MS Access, Text files and Excel files, as well as having and ODBC driver that lets you connect to many other databases. Now its even easier to manage a large number of database connections for multiple database types- keep all those servers and connections at your fingertips when you are combining all that data!

Data profiling in an affordable, graphical, visual environment

Find out why people are loving the combination of an ETL and a Data profiling tool in one- using Datamartist not just for data profiling, but for data migration, data quality audits, and ad hoc datamart creation. Try the free trial today.

Check out the two minute whirlwind tour video.

What's new in V1.3

Just a few of the great new things you'll find in this version, along with all the usual visual, drill downable data profiling and transforming goodness;

Value distribution data profiling

Quickly analyse the distribution of numeric values zooming in on questionable distribution areas, exploring visually, going from visual to drilling down into the underlying rows and back instantly. You can check it out in this video.

Automated data profiling

We've added a dedicated data profiling block to the block library of the professional edition that lets you sample as many points of the data canvas as you want, and write time-stamped data profiling results. The result is that not only can you profile your data automatically, you can analyse trends, and use the data profiling stream from Datamartist to create external dashboards and data quality alerts.

Enhanced data format rule editing

The professional edition also boasts an enhanced ability to directly edit data format analysis rules. By defining a series of simple rules, you can analyse the data format of your data quickly and easily. Are there letters where you should only have numbers? Are the product codes properly formed?

Regular Expression support (REGEX)

Datamartist professional now has a REGEX function that allows you to include regular expressions anywhere that expressions are used. This means you can use them in calculation blocks for transforms, segmentation tests, and filters. This gives you a powerful new tool to assess and profile your data, either writing your own, or using one of the many expressions available.

For a more detailed look at Datamartist, you can browse the documentation, or better yet, download the free trial now, and see what Datamartist can do with your data.

The currently released version 1.2.6 is of course also still available, but for those who don't mind risking a bug or two (we need your help in killing the last few) the Beta gives you a sneak peak at the new version and a bunch of new features that have been introduced to the Professional edition.

Whats in it? Lots of data profiling goodies;

Value distribution profiling

One of the important additions to the data profiling capabilities in Datamartist Pro is the value distribution explorer. This powerful functionality analyzes numeric fields and provides a value distribution graph (equal size buckets for row counts based on value) that lets you zoom in and out, and drill down into the rows to understand your data and spot any suspicious values at a glance.

Support for Regular expressions for pattern matching

The new version has added a very powerful function to the function library in the professional edition;

REGEX(text,regex expression)

With this function, it is now possible to use regular expressions (regex expressions) to evaluate if string values conform to desired data formats. Regular expressions are widely used for data quality testing, and there are lots of them available out there.

Custom data format rules

In V1.3.0 professional edition, it is now possible to add a series of custom rules for character mapping, creating much more flexibility in the data profiling tool. For example, its possible to map numbers, letters, punctuation characters, or any combination of them.

Data profiling block

In the professional edition, the category of data quality blocks has been added, and the first block released is the data profiler block.

This lets data profiling results themselves be used within your data canvas, as well as written out to files or database tables, with time stamps- enabling data profiling automation and tracking. It is now possible to define, calculate and track data profiling metrics automatically, sampling data at a sample frequency that lets you understand how your data quality is evolving.

We're excited about what can be done with this first data quality block- and we are sure our professional edition users will be tracking their data quality like never before.

I'll be blogging about all the new features in the coming days and weeks- I'm certain you'll find lots of things to like in the new version- give it a go.

As always with our betas, we love feedback, and we'll be giving some free PRO licenses away to our most active Beta testers.

• Easy to install
• Easy to use
• ETL features and data profiling capability

Avoid using the wrong tool for the job

Enterprise ETL tools (Extract Transform and Load) are very powerful but often extremely difficult to use.

• expensive, particularly if multiple environments are needed
• require server infrastructure, configuration and setup.
• require expensive developers who have been trained in the specific programming language of each particular vendors tool.
• designed for performance and data volume, not ease of use.

Obviously they have their time and place, but when you want fast, visual access to your data, you end up getting slowed down by expensive ETL server overkill.

A better choice- the visual, clean ETL tool

Datamartist is designed to let you extract data from multiple sources, and then mix it, match it, transform it, and understand it.

It uses a visual block and connector model, with the concept of "Data canvases" that let you easily manage and simplify complex data transformations. But unlike many overly complex ETL tools, Datamartist provides visual, configurable blocks, rather than requiring code.

Easy to install

Datamartist installs in minutes, and runs on your desktop, giving you control of your data, and what you need to do. Don't configure servers, don't worry about installing the right version of Java, don't spend hours searching wikis and forums and tweaking config files. Just download it, single step install it, and use it.

It makes it easy for you to take a snapshot of the data you need- locally with cut and paste or drag and drop from files, and locally or remotely with native connections to SQL Server, Oracle, MySql and MS Access, and pretty much anything else via ODBC.

And since the Datamartist data transformation engine can be run from the command line or scripted, it can also be automated to implement ETL tasks running on a windows server.

Speed up data delivery, reduce cost.

Datamartist provides a flexible, simple to use ETL environment that will let you shorten your time to delivery significantly for a wide range of data transformation tasks.

• Deliver small and medium sized data transformation tasks more quickly
• Build rapid prototypes and proofs of concepts
• Automate data profiling and data quality monitoring

Give the Datamartist ETL Tool a try

You can download the Datamartist trial and be up and running in minutes. You don't even have to register- and you will have full access to a fully functioning version of Datamartist to try out this simple, visual ETL tool on your own data.

We're also very excited about V1.3.0, currently in private beta. If you'd like to participate in the public beta, drop me a line at "beta at datamartist.com", and we'll send you a link when that download is available.

In this version, we've introduced a Standard and Pro edition, letting customers get the features they need at the right price.

• Datamartist Standard: $349
• Datamartist Professional: $745

A comparison of the feature sets explains the details.

Whats new in V1.2

Data source import enhancements

• Ability to cut and paste between Excel, Text files, the Datamartist canvas and any Datamartist data viewer.
• New integrated data source repository with drag and drop to canvas.
• SQL Editor to allow the creation of SQL queries to get data from databases.

Running Datamartist canvases automatically

Now that Datamartist can be run from the command line, it is possible to schedule datamartist transforms- even running it on a Windows server. Details about the logging and options are here.

Edit Internal data sets.

The addition of fully editable internal data sets that are stored within the DMC file itself gives a powerful new ability to create "What if" type scenarios. Imagine you want to see the effect of changing the sales regions slightly- just copy and paste the existing from a data viewer onto the canvas- that gives you an internal data set block with that data in it- now you can add a column "New Region" or rename the column, then edit some values, join it back into the original data with a join block, and be trying different scenarios in no time.

We're excited about this new release, and thanks to all our customers and testers for their feedback- we're glad to be incorporating some of those great ideas into the product.

If you haven't tried Datamartist yet, this is the perfect time, and now with two editions to choose from you can get the features you need at the right price.

Tableau has received a fair amount of positive response from the visualization world. Even Stephen Few, who isn't shy to point out when visualizations are straying from the straight and narrow has been supportive of Tableau from the start.

We're excited about this new access to such a great data visualization tool because we know that people who do visualization have to transform their data- and if users of Tableau want a flexible, visual data transformation tool, the Datamartist tool is an obvious choice.

Free Tableau vs Professional version

There are of course some limitations with the free version of Tableau, in comparison with the full featured professional version ($1600 USD per seat.)

• Data import capabilities - Only MS Access, Excel and Text files
• 100 000 row Limit per table
• 50 Mb Limit per organization for the web server.

None of these limitations will stop you from making some pretty fantastic visualizations as long as your final summarized data set fits within the limits and you put it in the right format. But it does mean that to use this version of Tableau, you need to use another tool to get those large data sets summarized. This is probably something you are doing anyway, because there is almost always some data cleanup to do.

Get your data sets ready- there are going to be some beautiful viz getting made

The trick with visualization of course, is that you need data to visualize. With tools like Tableau, as good as they are at making the pictures, you have to get the data set to them first- and we all know how many pre-formatted, all is well, no data quality issues data sets there are lying around the real world (Hint: none.).

But I'll tell you, once you do have the data, what a fantastic bit of interactive web based visualization Tableau can do.

Often, people use a combination of MS Excel and Access to create the datasets that they want, then connect to the Access database with Tableau. Of course, we suggest you try Datamartist instead. Datamartist and Tableau are a powerful combination- first, using Datamartist pull data from mutliple sources (Datamartist loads out of SQL Server, Oracle, MySQL, and at a license price much less than Tableau Professional).

Datamartist lets you join tables visually (using a venn diagram interface that we're proud of), segment data using rule sets, summarize millions of rows if needed, and generally parse and transform with an easy to use calculation engine. Once you have the data where you want it, export it easily to an Access database, and let Tableau Public generate the visualizations that you need.

I'm excited about Tableau's decision to make its power available in this public version. I intend to do some serious data crunching with the Datamartist Tool, followed by some interactive visualizations with Tableau Public. The beauty of the Tableau Public setup is that I can then publish the visualizations right here in the blog, and highlight what the combination of these two tools can do.

Joins can be said to be INNER or OUTER joins, and the two tables involved are referred to as LEFT and RIGHT. By combining these two concepts you get all the various types of joins in join land: Inner, left outer, right outer, and the full outer join.

Tables used for SQL Examples

In the screen shots I've configured Datamartist to only show the name columns to save space. The SQL code shown is "Select *" so it will return all the columns. You can see that in the Datamartist tool the type of join is selected by just checking the parts of the venn diagram that contain the rows you want.

1) Inner Join SQL Example

select * from dbo.Students S INNER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

2) Left Outer Join SQL Example

select * from dbo.Students S LEFT OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

4) Full Outer Join SQL Example

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

5) SQL example for just getting the rows that don't join

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID where A.Advisor_ID is null or S.Student_ID is null

6) SQL example for just rows from one table that don't join

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID where A.Advisor_ID is null

But what about the duplicate row thing?

Now, since in this case we had a simple one to one relationship, the number of rows that were returned made the venn diagrams make sense, and add up pretty normally with table one and two.

What happens if the data in the tables are not a simple one to one relationship? What happens if we add one duplicate advisor with the same ID, but a different name?

A join will create a row for every combination of rows that join together. So if there are two advisors with the same key, for every student record that has that key, you will have two rows in the inner part of the join. The advisor duplicate makes duplicate student records for every student with that advisor.

You can see how this could add up to a lot of extra rows. The number of rows is the product of the two sets of joining rows. If the tables get big, just a few duplicates will cause the results of a join to be much larger than the total number of rows in the input tables- this is something you have to watch very carefully when joining- check your row counts.

So there you have it. If you want to try joining tables with the Datamartist tool- give it a try. It's a super fast install, and you'll be joining like a pro in no time.

Thanks to Geoffrey Wiseman who covered the event on his blog Tech scene, you can watch the video of the presentation.

It was lots of fun, and we got some great feedback. A few people came up after and used the term "Excel Hell" to describe some of their previous data transformation experiences, and they were right into what Datamartist could do.

Note: The event was held in an auditorium in one of the Rogers offices (which explains the Rogers logo on the podium in the photos)- Datamartist has no affiliation with Rogers of course, but we're grateful for all the support they give the Toronto startup community through sponsoring Democamp and providing such a great space for the sold out event.

Sneak preview

We've still got a heavy development cycle in December, but we're excited about the release we're targeting to get out the door early in the new year. Because we worked to put together a specific build for this demo, and it's pretty stable, I've made the decision to make this beta publicly available- if you go to the download page you'll find a link at the bottom that will let you download the installer. As with anything beta, it might still have some bugs in there- but overall you'll find some key new functionality that went over very well here in Toronto:

• Drag and drop for file import- just drag flat files or excel files onto the Datamartist canvas, and the blocks are instantly generated, ready to go.
• Copy and paste data - Copying a range in excel, then right clicking or pressing CTRL-V in Datamartist generates a new kind of block called a "Internal data set". More about these blocks soon, as we have lots of plans for them in this upcoming release.
• Copy and paste subsets from data viewer- Now you can also copy and paste data internally- click on a stub of a block, select only the rows, or only the columns, or even the cells that you want, press CTRL-C then right click on the canvas and you can paste that data into a new internal data set block.

Of course, copy and paste has some limits in terms of data sizes, but it is very useful for reference sets and can still handle pretty good sized sets.

The whole drive in continuing to enhance this type of functionality is about making an environment where its easy to move data around, in and out, and transform it- being able to explore the data and get it right where you want it in an intuitive, flexible environment. Stay tuned, lots more to come!

This version is fully backward compatible with all previous versions of Datamartist, and we encourage all our customers to upgrade.

This version of Datamartist brings with it a number of enhancements, some of the highlights:

Native data base import/export for Oracle and MySQL Enterprise Server

We're very happy to have added two more very significant database vendors to the natively supported list, and expect more in the future. In addition to adding the connectivity, we've also made enhancements to the general import interface, particularly in the selection and browsing of tables from large databases. Check it out.

A new Join block that lets you join inner, outer, right and left to your hearts content;

We are particularly proud of this block, because it gives a clean, clear visual view into one of the most important data analysis functions, the table join. You can specify the scope of the join easily using the venn diagram, and can drill down into the various result sets and see whats joining and whats not.

Enhanced copy and paste.

We've made almost all the data grids selectable and copy enabled, so you'll find that you can easily grab a value, a row, a columns, or any combination, copy it, and then paste it into Excel or other applications. We've been loving the flexibility this gives us, and hope you like it too.

We're always looking for feedback from our users, so let us know what you think.