The trick with public data like this is that often (ok almost ALWAYS) either data is missing, or the codes don't match up. In this case, the country descriptions from various data sets I want to use don't match- and different data sets have different holes (i.e. not all datasets include data for all countries). Finally, some data sets have a different definition of a country- for example, they break out places like "British Indian Ocean Territories" that need to get rolled up in the UK numbers.

Country statistics data such as GDP, GNI and Population were also incorporated to provide dimensions to carry the analysis out on. The raw trade data was obtained from Industry Canada's "Trade Data Online" (http://www.ic.gc.ca/sc_mrkti/tdst/tdo/tdo.php?lang=30&productType=HS6). The World Bank was the source of the country statistics data (http://data.worldbank.org/indicator). A zip file containing the raw data, as well as the Datamartist data transformation .dmc file is provided at the bottom of this post.

I started by transforming the country statistics data. The raw data included information on GDP, GNI, Total Population and Urban Population. A screenshot of the Datamartist canvas for the first portion of this data transformation is provided below.As seen above, the first step in the data transformation involved importing the four excel data files. During this import, columns with data not relevant to the year 2009 were filtered out and zeros were inserted into any null data rows, signifying that data for that row was not available. A series of data "Join" functions were then carried out to create one data file containing all of the country statistics information. Upon completion of joining these data files," a "Calculation" block was utilized to replace any null data values resulting from the data join with zero's. Finally, the country statistics information was joined with a country cross reference list. Basically, this join standardizes all of the country names.

The second part of the country statistics data transformation focused on segmenting the data, as shown in the Datamartist canvas screenshot below.Before the data could be segmented, it was summarized so that there was only one data row for each standardized country name. A "Calculation" block was then added to calculate the GDP per Capita, GNI per Capita and Urban Population Percentage using the Population data. With these calculations complete, a series of "Segment" blocks were added to the canvas. The "Segment" blocks are extremely useful because they add an additional column to the data set which is populated according to a set of segmentation rules defined by the user. In this example, the "Segment" block was used to segment the GDP per Capita, GNI per Capita, Urban Population Percentage and Population data. The segmentation rules for the Population "Segment" block are shown in the screenshot on the left.

A similar set of data transformations was also carried out on the Canadian Import and Export Trade data. As seen in the Datamartist canvas screenshot, the Canadian Import and Export Trade data was imported, joined and null data values were replaced with zeros. The country names were then standardized and summarized so that the Canadian Import and Export Trade data could be joined with the Country Statistics data.

With all of the raw data transformed into a suitable format, a final set of data transformations were carried out to create a single text file. This text file was then exported so that QlikView could be used to analyze the data. A screenshot of the Datamartist canvas for this final set of data transformations is shown below. In this final set of data transformations, the "Star Schema" block was used first. The "Star Schema" block is a handy data transformation tool because it allows numerous data join operations to be carried out simultaneously. It was used to combine the Country Statistics data and Canadian Import and Export Trade data with data defining a country's geographical region. A screenshot of the "Star Schema" block configuration window is shown to the left. The joined data was then put through a "Calculation" block one last time to eliminate any null data values. Finally, the transformed data was exported as a text file so that it could be analyzed in QlikView.

The transformed data was then imported into QlikView and a dashboard was created to analyze the data with. QlikView is a great data analysis tool because it allows data to be filtered and visualized very efficiently. In this example, I made a dashboard that allows Canada Import and Export Trade data to be visualized using the Country Statistics Data segments created using the Datamartist software as filters. A screenshot of the dashboard with no filters applied is shown to the right. I am now going to show a series of screenshots with different data filters applied. To start off, I want to see the countries that Canada Exports the most goods to. To do this, I just dragged a box over the largest bars on the export graph as seen below. Once I finished making the data selection box on the graph, I released the mouse and QlikView automatically zoomed in on the area I selected. In addition, the upper right table in the dashboard updates as well. The image below shows the results. As seen in the image, Canada's biggest export trade partners in 2009 were the United States, the United Kingdom and China. As a second example, I will filter the data using the data segments created in Datamartist. If I click on "Asia" in the "Region" box then only data from the countries in Asia is shown in the table and graphs. Furthermore, the segments in the other filter boxes (GDP per Capita, GNI per Capita, etc) updates to the region selection as well. It does this by highlighting the data filter segments that are valid for the "Asia" region in white. For example, in the "GDP per Capita" box, all data segments are valid except for the "GDP > $100 Thousand" segment. A screenshot of QlikView with the "Asia" region filter on is shown below. I can further filter the data by clicking on any other data filter segments that are white. As an example, if I select "$1 Thousand - $5 Thousand" in the "GDP per Capita" box and "40% - 60%" in the "Urban Population Percentage" box, the graphs and table update again. In this instance, the only countries that meet these filter requirements are China, Georgia and Mongolia. A QlikView screenshot with all three of the filters chosen is shown below.

Try it out yourself with the free trial

You can give Datamartist a try with this data, just signup and download the free trial, and then download a zip file will all the data, and the example Datamartist file.

Just extract all the files in the above ZIP file into the "My Datamartist" folder that the Datamartist trial will create when you run it, and open the "World Trading Example.DMC" file with Datamartist.

You'll find that Datamartist gives you a powerful, visual way to transform data from lots of places, and get it ready for great visualization tools like Qlikview in a step by step, clean, repeatable way.

On top of that, datamartist can be automated- so if you have data transformations you need to run on a schedule, you can design them in a graphical environment, test them, and then have them run automatically.

• Easy to install
• Easy to use
• ETL features and data profiling capability

Avoid using the wrong tool for the job

Enterprise ETL tools (Extract Transform and Load) are very powerful but often extremely difficult to use.

• expensive, particularly if multiple environments are needed
• require server infrastructure, configuration and setup.
• require expensive developers who have been trained in the specific programming language of each particular vendors tool.
• designed for performance and data volume, not ease of use.

Obviously they have their time and place, but when you want fast, visual access to your data, you end up getting slowed down by expensive ETL server overkill.

A better choice- the visual, clean ETL tool

Datamartist is designed to let you extract data from multiple sources, and then mix it, match it, transform it, and understand it.

It uses a visual block and connector model, with the concept of "Data canvases" that let you easily manage and simplify complex data transformations. But unlike many overly complex ETL tools, Datamartist provides visual, configurable blocks, rather than requiring code.

Easy to install

Datamartist installs in minutes, and runs on your desktop, giving you control of your data, and what you need to do. Don't configure servers, don't worry about installing the right version of Java, don't spend hours searching wikis and forums and tweaking config files. Just download it, single step install it, and use it.

It makes it easy for you to take a snapshot of the data you need- locally with cut and paste or drag and drop from files, and locally or remotely with native connections to SQL Server, Oracle, MySql and MS Access, and pretty much anything else via ODBC.

And since the Datamartist data transformation engine can be run from the command line or scripted, it can also be automated to implement ETL tasks running on a windows server.

Speed up data delivery, reduce cost.

Datamartist provides a flexible, simple to use ETL environment that will let you shorten your time to delivery significantly for a wide range of data transformation tasks.

• Deliver small and medium sized data transformation tasks more quickly
• Build rapid prototypes and proofs of concepts
• Automate data profiling and data quality monitoring

Give the Datamartist ETL Tool a try

You can download the Datamartist trial and be up and running in minutes. You don't even have to register- and you will have full access to a fully functioning version of Datamartist to try out this simple, visual ETL tool on your own data.

We're also very excited about V1.3.0, currently in private beta. If you'd like to participate in the public beta, drop me a line at "beta at datamartist.com", and we'll send you a link when that download is available.

In this version, we've introduced a Standard and Pro edition, letting customers get the features they need at the right price.

• Datamartist Standard: $349
• Datamartist Professional: $745

A comparison of the feature sets explains the details.

Whats new in V1.2

Data source import enhancements

• Ability to cut and paste between Excel, Text files, the Datamartist canvas and any Datamartist data viewer.
• New integrated data source repository with drag and drop to canvas.
• SQL Editor to allow the creation of SQL queries to get data from databases.

Running Datamartist canvases automatically

Now that Datamartist can be run from the command line, it is possible to schedule datamartist transforms- even running it on a Windows server. Details about the logging and options are here.

Edit Internal data sets.

The addition of fully editable internal data sets that are stored within the DMC file itself gives a powerful new ability to create "What if" type scenarios. Imagine you want to see the effect of changing the sales regions slightly- just copy and paste the existing from a data viewer onto the canvas- that gives you an internal data set block with that data in it- now you can add a column "New Region" or rename the column, then edit some values, join it back into the original data with a join block, and be trying different scenarios in no time.

We're excited about this new release, and thanks to all our customers and testers for their feedback- we're glad to be incorporating some of those great ideas into the product.

If you haven't tried Datamartist yet, this is the perfect time, and now with two editions to choose from you can get the features you need at the right price.

Joins can be said to be INNER or OUTER joins, and the two tables involved are referred to as LEFT and RIGHT. By combining these two concepts you get all the various types of joins in join land: Inner, left outer, right outer, and the full outer join.

Tables used for SQL Examples

In the screen shots I've configured Datamartist to only show the name columns to save space. The SQL code shown is "Select *" so it will return all the columns. You can see that in the Datamartist tool the type of join is selected by just checking the parts of the venn diagram that contain the rows you want.

1) Inner Join SQL Example

select * from dbo.Students S INNER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

2) Left Outer Join SQL Example

select * from dbo.Students S LEFT OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

4) Full Outer Join SQL Example

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID

5) SQL example for just getting the rows that don't join

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID where A.Advisor_ID is null or S.Student_ID is null

6) SQL example for just rows from one table that don't join

select * from dbo.Students S FULL OUTER JOIN dbo.Advisors A ON S.Advisor_ID=A.Advisor_ID where A.Advisor_ID is null

But what about the duplicate row thing?

Now, since in this case we had a simple one to one relationship, the number of rows that were returned made the venn diagrams make sense, and add up pretty normally with table one and two.

What happens if the data in the tables are not a simple one to one relationship? What happens if we add one duplicate advisor with the same ID, but a different name?

A join will create a row for every combination of rows that join together. So if there are two advisors with the same key, for every student record that has that key, you will have two rows in the inner part of the join. The advisor duplicate makes duplicate student records for every student with that advisor.

You can see how this could add up to a lot of extra rows. The number of rows is the product of the two sets of joining rows. If the tables get big, just a few duplicates will cause the results of a join to be much larger than the total number of rows in the input tables- this is something you have to watch very carefully when joining- check your row counts.

So there you have it. If you want to try joining tables with the Datamartist tool- give it a try. It's a super fast install, and you'll be joining like a pro in no time.

But there is a way to get a handle on the tricky parts of your data warehouse scope, and to reduce your projects overall cost.

The major cost component of any data warehouse project is the Extract Transform and Load (ETL) development. Obviously every project is slightly different, but in my experience ETL will often make up in the order of 70% of the development cost. One of the drivers of this cost is the relatively high priced ETL development resources required. In the markets where I've hired resources, an ETL developer will often demand a 30-40% higher hourly rate than a business intelligence report writer, for example.

Making ETL prototypes will give you insights that can reduce cost by shortening the ETL development process and making the optimum use of those highly talented and expensive ETL resources.

What affects the cost and complexity of ETL jobs?

For any given scope, the following will have a large impact on the number and complexity of ETL jobs and therefore their cost.

1. The number of different data sources involved.
2. The consistency in terms of master data definitions between systems.
3. The level of data quality in the systems.

Ideally, you want to get a good handle on these three things before you hire all the ETL developers, and be confident that you are going to satisfy the users needs before millions of dollars are spent on Extract Transform and Load (ETL) jobs and business intelligence reports.

One part of the preparation needed to do this can be the creation of a proof of concept or mockup of key parts of the data warehouse ETL deliverable.

Now, there are mockups, there are prototypes, and there are "first versions". The the most effective approach is to create a mockup or prototype that;

• Goes just deep enough into the data to:
  • Establish all data sources that will be required
  • Gives a high level audit of their master data and data quality
• Provides enough output that:
  • End users can be supplied with example reports or cubes to get hands on
  • The functional scope can be locked down with confidence on all sides.

The goal of a data warehouse prototype is to learn about the underlying data, and to be able to try different data transformation techniques and approaches on the real data. The goal is not to make the finished product, nor to deliver actually usable reports to end users, although it may be to generate an example result for users to validate.

An example might be to create a prototype to calculate total sales by segment for a period under a new customer segmentation. This would identify if the segmentation rules that have been suggested actually result in the expected segementation of sales data, and if the fields involved are complete and correctly populated in the source systems.

A prototype should focus on the dimensions and data sources that are expected to be the most difficult, and involve multi-source integration. Don't spend time prototyping the easy stuff.

When you are making a prototype remember its a one-time development. Manual steps and doing some "data cleaning by hand" are perfectly reasonable- its what you learn from the prototype, not how you learn it that is important. Take a snapshot, or a sample of the various tables and put them in a sandbox environment where you can manipulate them quickly and easily.

The whole point is to move quickly, get lots of feedback from users, and be able to avoid unpleasant discoveries during the actual data warehouse development.

If you find a data quality issue, and it's a tough one, then just remove those rows and continue on- remember you don't have to solve all the problems in the prototype- you need to identify them. Be open with your users about what the exercise is about- and that it is a very rough pass, and a mockup.

How much could this impact cost?

If you can identify issues during the prototype then you can solve them before all the ETL development resources are brought onto the project.

If you do not do a prototype, and find a data quality issue that requires some back and forth with the business, every week of delay will probably represent thousands or tens of thousands of dollars, with the project team waiting on the resolution before being able to resume coding the ETL jobs in question.

So in summary, making prototypes will:

• Reduce the risk of scope creep because users have actually seen and "touched" a mockup of the final output.
• Reduce the amount of rework in ETL code because different data transformation approaches can be tested early.
• Reduce the risk of the expensive ETL development phase of the project slipping due to unknown data quality issues.

The right tool for ETL Prototypes.

Often prototypes are built in a combination of Excel, MS Access or other databases. These tools can work, but excel has serious issues handling larger data sets, and database development is often cumbersome- the idea is to make a prototype, not actually build the SQL code. Things like different data types, field formats, column naming rules etc. between different source databases often frustrate attempts to do something quickly.

Obviously another option is the enterprise ETL tools themselves- but the cost, complexity and overhead of these tools again makes them better suited to the production system- not a quick mockup or rapid prototype.

What you need to make an ETL prototype is an easy to use ETL tool that provides the basic type of functionality and graphical user interface of high end ETL tools, but also allows a more flexible treatment of data types, all with the ability to pull data from multiple sources, including more informal sources like Excel spreadsheets.

The Datamartist tool was created to provide exactly such a data scratchpad, ideal for rapid prototyping data transformations. It lets you profile your data and build data transformations using a visual, block and connector interface. But it represents a clear, focused and easy to use ETL tool, without all the feature bloat, cost and server configuration required by many expensive enterprise ETL solutions.

Download the free trial, and see for yourself.

I believe in an approach that firmly places people in the center of a new way of looking at the data analysis process.

In the past, “big business intelligence” created an infrastructure heavy, highly centralised and technology focused approach to getting data from source systems into reports in the hands of the users. Under this regime, users were not to be trusted with raw data, but were given tightly controlled, managed and aggregated reports in order to protect the “single version of the truth”.

The theory and practice were tightly defined, and had been honed over decades of business intelligence and data warehouse orthodoxy. Giving raw data to end users would lead to chaos. Letting end users define new ways to look at the data would corrupt the master data, and lead to everyone looking at something different.

You can guess the sort of response this "don't give them the raw data" approach gets from capable, curious people that want to get down to some real analysis.

But to be fair you can see why these concerns are thought to be well founded. Almost every large enterprise is awash in a sea of excel files and a tangle of links and formulas. Excel is a wonderful tool, but it only offers the illusion of solving the data transformation problem. It is a much better reporting/dashboard tool than an ETL. (Although in the right hands it can do remarkable things.)

And this is the true state of affairs now. When the “official” system does not provide the answers that the business needs the people who need to make decisions get the data anyway, and they do it themselves. They do it in excel, they take night courses in Structured Query Language (SQL) they hire consultants (or even summer students) to build rogue data bases that they run on servers hidden under desks to get at the answers they need.

It is easy for the data warehouse theorists to highlight the clear issues with "spreadmarts" and "shadow systems".

But we need to be pragmatic. The reality of building a centralized structure that imposes strict formal rules and change management processes is that often while it does ensure that there is only one version of the truth, it is a version of the truth that no one can use because it has been so formalized, aggregated, compromised and delayed that by the time it is delivered the pressing business questions have changed and meaning has been expunged. The data warehouse becomes reporting rather than analysis.

Its clear that enterprises need this kind of reporting- I'm not advocating abandoning the existing approach- but augmenting it. Up till now, the solution has often been "more of the same".

The regime decided that the solution was to add more technology to the central systems, increase enforcement, and search out and repress all the dissident data manipulators. The data resistance was forced to go underground, to hide their spreadsheets, to outwardly appear to be following the official line.

It is very true that there are some risks in allowing people to analyze their own data, but there is also a reward. There are a small group of people who love data, who understand the business questions, who work to tease insight out of a steaming pile of raw data and can find things that are game changing. Massive, formal, designed by committee data warehouses can deliver a powerful and useful view of things, but they rarely offer flashes of insight. When they do, it is often during the design and discovery process- rarely by users using the system after it has gone live.

The Datamartist tool has been built based on the belief that both formal, centralized systems AND local, personal data transformation have a place in the architecture and that both should be official places.

People can be trusted with the data. In fact I think for an organisation to truly be successful at mastering its information, they have to be.

We have to realize that we can't allow our obsession with the quest for a single version of the truth to turn us into totalitarian regimes, certain that OUR truth is THE truth, and that messing around with the data is by its very nature subversive and dangerous.

Data to the people.

So the idea of self serve business intelligence is a wonderful idea- the problem is that its not all about pretty graphs and fancy web user interfaces. You need to somehow design the data model so that every possible report that users dream up is possible- or move so much data modeling functionality into the report writer that it ends up looking more like a data transformation tool, yet is easy to use. There are "new" techniques that are getting lots of discussion- columnar databases are one, and certainly they provide interesting techniques, but only IF you've got meaningful data.

Here's something to try- call up your favorite Business Intelligence vendor, and ask for a demo. It will be wonderful, it will be clear, easy, and simple- and it will be done on a set of data that was made by someone who knew exactly what the demo was going to be.

Real world data is messy, and it often does not follow simple rules. As a result, it takes significant work to build analysis- and to have a system that is ready for ANY analysis that any user might think up at the moment is non-trivial.

But its a worthy pursuit. And I think there are three key fronts in this battle;

1. Reporting and Analysis front end tools
2. Data transformation and integration tools
3. Fixing the source systems.

All the big players have integrated suites of products that perform the functions of the first two categories- IBM/Cognos, Oracle, SAP.

There are also lots of very interesting new tools in the first category- Tableau is one that gets a lot of buzz. Of course, the king of graphing and dashboarding is still the spreadsheet, and Excel has the crown.

There are lots of tools for the IT department in the second category. Datamartist is a unique new tool for the end user in this area- a self-serve desktop tool for data transformation. It is a tool that allows users to quickly transform data sets to experiment and create new analysis that can then be queried and viewed using the tools in the first category.

But in the end, I think the number one limiter on achieving the dream of self-serve business intelligence will be getting a handle on the quality of the data in the source systems. Garbage in means garbage out- and the last two layers shouldn't have to tie themselves in knots trying to fix issues that are generated in the transactional systems.

But they do tie themselves in knots- and it is those knots that stop users from more freely and accessing their data using an intuitive and almost brainstorming approach- which in the end is the goal of self serve.

Unless the underlying data in the source systems is quality controlled, and designed to capture the information that is critical for analysts, then the Business Intelligence layer will have to work too hard to "fix" the data, keeping a large IT team busy writing code, and there's nothing self serve about that.

In the end, the data quality issue is often what makes data warehouses and data marts so expensive to build- and drives users to spreadsheets or even databases (microsoft access is a common one). This is what I call "self serve data transformation"- and this is what Datamartist does- if you're frustrated with the access you have to your data- download it and give it a try.

Check out the first four parts of this series (1,2 , 3 and 4) where we created an example data mart, with some fictitious purchasing data.

The final step is to join the dimensions we have created to the fact table. To do this, we connect up the two dimensions (Vendor and Item) to the Join block and connect an export block to the output. What has in effect been created is a complete Extract, Transform Load (ETL) and the final star schema join.

(If thats a bit hard to read- click on the image to see the full size screen shot.)

With the generated data set I used for this example, summarizing the data to yearly totals but keeping all the detail on Vendor and Item causes the roughly 4 million row raw data file to be reduced to around 800 thousand rows. (This summarizing was done on another canvas- although it could have been done on this canvas just as easily).

This data mart, with 800 k rows and two dimensions of about three thousand members each took my laptop about a minute and 45 seconds to solve, and save to a 360 Mb text file out.

Of course, by summarizing or filtering (just add blocks) analysis subsets could easily be exported directly to Excel, managing the data volumes involved, and letting you create the graphs, dashboards and reports that you need.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Now we need to create a Vendor dimension table and join it to this fact table to determine who our big vendors are.

In Datamartist it is a simple task to create this vendor dimension. As always we use blocks and connect them together. We define a dimension by using a reference definition block. All we have to do to configure the reference block is to specify which columns uniquely define the dimension (or almost uniquely, Datamartist will resolve duplicate keys using a majority/first rule set for you if you have some data glitches).

We start with an import block that brings in the Vendor master text file, then we define the reference by specifying "Vendor_ID" as the key. These first two blocks look like this:

Then we join it to the fact table we created in part two of this series with a join block. This means that now instead of just the vendor ID number that was in the fact table, we have the name, and address for the vendor in our mini star schema.

And finally we put a summarize block after that to total up all the monthly values for each vendor, and we export to excel. This is what the canvas looks like:

After we do this, we grab the excel file Datamartist just created for us, do a quick sort, and come up with a list of Acme's top ten suppliers. Feeling pretty good about ourselves, we do a review with the head of purchasing.

"Where's Mega brothers?" she says with a frown "I think your data is screwy- no way that Mega brothers didn't make the top ten- we spend a fortune on railways, and a lot of our freight goes with the Mega Brothers Rail company. Of course it is probably entered under different vendors, each location works with the office local to them... But we've got to view them as a single vendor in the data mart- you can do that right?"

Fixing Duplicate Rows

Having to deal with duplicate data is a very common issue in any type of data analysis. So, back to the canvas. By simply adding a de-duplicate block to our Vendor dimension table (after the Reference block, and before the join) we can find and resolve the Mega Brothers duplicates.
We just use the filter to find the records- (Easy to do, looking for "Mega" "rail" "brothers" etc. and we map them to a single instance.) This is the filter control that lets us find and tag the duplicates:

As we tag them, they show up in the mapper, which lets us see which duplicate records we have eliminated for the dimension. We run the canvas again, and this time, sure enough, Mega Brothers Rail is in our top ten. But even though the head of purchasing knew it was a lot, this is actually the first time she's seen the number. "Wow. I've got to give them a call- can you give me that in an Excel spreadsheet?"

Stay tuned, more to come as we go further into Datamartist's ability to segment, filter and organize large data sets.

If you want to see the interface in action watch our first Tutorial Video. Or just get right to it with your own data- download the free trial now- there is no registration required, and it installs in minutes.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

• Item_Master.txt - this holds all the items that Acme buys
• Vendor_Master.txt - this holds a list of all the vendors, with information such as their address
• PO_Detail.txt - this is the huge data set, all the purchase order data for the last four years

The Item and Vendor files aren't very big, but the PO_Detail is over 340 Mb, and it holds almost four million purchase order lines. Don't try to import it into Excel. Of course you need Excel 2007 to even try to import 4 million rows. In Excel 2003 it would take over sixty sheets and probably some VBA code to try it. I tried the import in Excel 2007- it takes 20 seconds just to tell me I'll have to go back to the text file import multiple times to do multiple imports onto separate sheets. It took almost two minutes to do the first million rows. Even once we have the data spread across four sheets it's not clear how to summarize millions of rows in excel easily.

Instead, let's use the Datamartist tool to manage this data set and generate one thats more useful.

The first analysis we will do will be on the Vendor dimension, to determine who Acme's big vendors are, and if we can negotiate some price reductions where we have leverage.

In Datamartist, very large files are not an issue because the tool can load in only preview data- this means that it's possible to look at a sampling of a few hundred thousand rows, and design the transformation before running it on the whole data set.

The PO Detail file has the columns shown- let's answer the question - "Who are our biggest suppliers"?
So which columns do we need? We probably want to have some sense of trends over time so we'll keep the order date, but summarize to Month, we'll keep the Vendor ID of course, and then we need to use the Quantity and Price fields to calculate the total amount spent. Then we want to write this summarized data into Excel to check it out.

To do this in Datamartist all it takes is four simple blocks; A Text import block to load in the PO_Detail.txt file, a calculate block to multiply QTY by PRICE, a Summarize block to do all the summarizing, and an Excel export block to generate the excel file;

Each block passes its result to the next block via the connectors, and the last block saves it to an excel file we've specified.

Defining the calculation uses standard spreadsheet functions- here's what the config area looks like;

And defining the summary is as simple as it looks- pick the columns you want, and select what kind of summary you want done.

We run it on a preview set of 100 thousand rows (takes about twelve seconds to run), and check the output.

It looks good, so we run on the whole 4 million rows;

About seven minutes later we have our result- an excel sheet with a manageable 130 thousand rows, total spend, by vendor, by month for four years;

Next up we need to create our vendor dimension, and join it to this mini fact table we have created. Stay tuned.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5