• Item_Master.txt - this holds all the items that Acme buys
• Vendor_Master.txt - this holds a list of all the vendors, with information such as their address
• PO_Detail.txt - this is the huge data set, all the purchase order data for the last four years

The Item and Vendor files aren't very big, but the PO_Detail is over 340 Mb, and it holds almost four million purchase order lines. Don't try to import it into Excel. Of course you need Excel 2007 to even try to import 4 million rows. In Excel 2003 it would take over sixty sheets and probably some VBA code to try it. I tried the import in Excel 2007- it takes 20 seconds just to tell me I'll have to go back to the text file import multiple times to do multiple imports onto separate sheets. It took almost two minutes to do the first million rows. Even once we have the data spread across four sheets it's not clear how to summarize millions of rows in excel easily.

Instead, let's use the Datamartist tool to manage this data set and generate one thats more useful.

The first analysis we will do will be on the Vendor dimension, to determine who Acme's big vendors are, and if we can negotiate some price reductions where we have leverage.

In Datamartist, very large files are not an issue because the tool can load in only preview data- this means that it's possible to look at a sampling of a few hundred thousand rows, and design the transformation before running it on the whole data set.

The PO Detail file has the columns shown- let's answer the question - "Who are our biggest suppliers"?
So which columns do we need? We probably want to have some sense of trends over time so we'll keep the order date, but summarize to Month, we'll keep the Vendor ID of course, and then we need to use the Quantity and Price fields to calculate the total amount spent. Then we want to write this summarized data into Excel to check it out.

To do this in Datamartist all it takes is four simple blocks; A Text import block to load in the PO_Detail.txt file, a calculate block to multiply QTY by PRICE, a Summarize block to do all the summarizing, and an Excel export block to generate the excel file;

Each block passes its result to the next block via the connectors, and the last block saves it to an excel file we've specified.

Defining the calculation uses standard spreadsheet functions- here's what the config area looks like;

And defining the summary is as simple as it looks- pick the columns you want, and select what kind of summary you want done.

We run it on a preview set of 100 thousand rows (takes about twelve seconds to run), and check the output.

It looks good, so we run on the whole 4 million rows;

About seven minutes later we have our result- an excel sheet with a manageable 130 thousand rows, total spend, by vendor, by month for four years;

Next up we need to create our vendor dimension, and join it to this mini fact table we have created. Stay tuned.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Data Size Limits in Excel

There are different types of limits-

1. The size in rows and columns the actual spreadsheet has.
2. Excel's (and your PC's) ability to crunch the numbers in a reasonable time. (RAM, CPU)
3. The size of the files involved and load and save times.

In Excel 2003, a spreadsheet has rows 1 to 65 536 and columns A to IV. This makes it a grid 256 X 65536. In Excel 2007 the spreadsheet is much, much larger, with rows from 1 to 1 048 576 and columns from A to XFD. (Making a grid 16384 X 1 048 576). Now before you get too excited about how much space you have in 2007, the reality is that limits number 2 and 3 define how you can actually use that space. But it is more and more is good.

So lets kick the tires on large data sets in Excel 2007. For these very informal tests I'm using a Quad-core workstation with 4Gb of RAM, so the results I get represent a best case compared to a typical laptop or desktop PC. First of all- putting a million rows of data in Excel 2007 (even a "narrow table" of only 3-4 columns) slows everything down. Delete a column, and you'll often see a 5-10 second freeze-up while excel churns away in the background- roughly the same amount of time needed to save the file. Plus, when I push it I've had it lock up on me a few times- requiring some Ctrl-Alt-Del action to kill it. Even a narrow table such as this makes the Excel file be at minimum 15-20 megabytes. For the particular text file I used, the .txt version was 9 Mb, the .xlsx file was double the size at 18 Mb. I added a few columns and the file quickly became 80 Mb.

Also, strangely, doing exactly the same thing multiple times results in very different times to complete- when I'm mentioning times its the average of 2-3 trials (see graph).

 All in all, although Excel 2007 can technically store a million rows, I'd advise against it. There are other reasons its a pain- scroll bars and page-up page-down don't scale well to 1M rows- its just hard to copy 250000 rows accurately- takes for ever to get to the end, and then you overshoot by a mile, and page up again forever to find it etc. etc. (And yes you can use the Go To command on the Home>Editing>Find and Select>Go to - but a model of ease its not.

I can tell you, however, that using all the other features on more reasonable data sets (up to say, 100 k rows), I LOVE what it can do in terms of analysis and reporting. Once you have the data in reasonable result sets, there is no better place to have it than in Excel if you want full control in my opinion. But how to get it there. Next posts: how to link to data in Access and build a mini personal data mart. We'll learn how to make a personal data mart given the currently available tools. (And you just know there will be some posts later where I show you how to do the same thing, but using Datamartist. ) Update: Datamartist now available. Download the tool now, and find a whole new way to transform and managed your data, including managing huge data imports into excel.