Let's face it, if you're considering Microsoft Access chances are what you need to get done is beyond what Excel does well, so you're looking for options. Its also likely that your IT department is unable or un-willing to help you out- this being even more likely as the recession reduces reporting budgets left, right and center.

Two of the key things that lead someone to search for a database solution are:

• Data Volume- More than a million rows and Excel becomes very difficult, even before that the performance suffers.
• Flexibility to Join Tables - Vlookup and VBA code only go so far- Access gives an easy way to make joins between tables, one of the powerful features of relational databases.

Now, the data volume is what it is- if you have millions and millions of rows, you need something to cut it down to size before you move it into your Excel spreadsheet.

On the other point, however, I can hear the Excel fans saying "now wait a minute, Excel can do that, I don't really need a database" and they are right. But they are almost always right- Excel can do almost anything. It does not mean, however that its the best tool for the job. Using Vlookup and VBA scripts to join up multiple tables is not my idea of a fun time. And even in Excel 2007 I find the pivot tables annoying and prone to break if I'm adding categories, moving data sets or heaven forbid changing number and order of columns.

Microsoft Access has a very nice interface for creating joins between tables, just a simple drag and drop between fields. The cross tab query capability is useful and good, and being a relational database it's more tolerant of changes to table structure because it's not messing with cell references.

"But", many who have used MS Access will say, "its pretty complex to learn, and even if I do start to get the query stuff down, it doesn't handle bad data well."

Bad data? Who has bad data? Isn't all data pristine, as intended, correctly formatted and accurate?

One of the huge differences between Excel and MS Access is that Excel is extremely flexible. (Probably more flexible than your auditor would like, but thats a different story). One source of Excels flexibility is its ability to accept different data types in the same column, and to allow editing of cells quickly. In Microsoft Access, for example, when it sees some variation it either discards the data or defaults to the data type "Text"- meaning now you can't perform the calculations you need to do on your data.

This illustrates one of the challenges people face in trying to use a database - databases are very strict on data types. Once you declare a data type for a column, if you import data into the table, the database will discard the values that do not conform to that data type. In Excel, you get cell errors if you try calculations but the original data is still there.

One of the powerful features of the Datamartist tool is the fact that it has an underlying database structure that provides flexibility on data types. Unlike MS Access and other databases, Datamartist can store dates, numbers, strings and booleans natively in a single column. (It does not convert to strings- it stores the full object). Take a look at this example:

In each individual row, Datamartist completes the calculation if possible. Datamartist is a database that gives you the freedom of a Spreadsheet. Of course, just like excel, if you ask for a calculation on a value that is meaningless you will get an error- but at the individual value- not a full row discard. This means that with messy data you can still work with it, bring it in, and fix it. In Access or another database, you can't even get it through the front door (or it defaults to text, making many calculations impossible).

This won't be the last time I compare these three tools- and the types of data structures and tasks each of them are most effective with.

In the mean time- Download Datamartist- see what I'm talking about with your own data.

There was a link posted on meta-filter today that sparked some lively discussion amongst the crowd. The Excel auto formating "feature" loves to scramble common genetic nomenclature.

It turns out that in the genetics field, common codes get converted to incorrect values regularly. One example given was the code for tumor suppressor "DEC1" which gets coverted to the date December 1. Another was the code "2310009E13" (apparently a "RIKEN clone identifier") - which would be converted to a number, 2.31E+19. I'm not a geneticist but I can just see how this wouldn't be helpful.

I checked these examples on Excel 2007, and sure enough, the default will make changes right at import that scramble the mentioned codes- no error, no notification, no problem. Of course Excel is perfectly capable of handling this data properly- the user needs to specify the field as text, and the conversions won't be done.
The key point brought up in the article (and is always true about excel spreadsheets) is not just that in this case the data gets corrupted but that depending on how carefully a user checks the error may not be detected.
If undetected, what decisions, conclusions and actions will be taken based on the incorrect information?

Excel is super powerful, and super useful, but we have to always remind ourselves to balance the ease of use with how critical our data is, and what the impact of errors might be. In the end, as with all computer use, we have to test, validate and test again at a level consistent with whatever use we are putting the data to.

• Item_Master.txt - this holds all the items that Acme buys
• Vendor_Master.txt - this holds a list of all the vendors, with information such as their address
• PO_Detail.txt - this is the huge data set, all the purchase order data for the last four years

The Item and Vendor files aren't very big, but the PO_Detail is over 340 Mb, and it holds almost four million purchase order lines. Don't try to import it into Excel. Of course you need Excel 2007 to even try to import 4 million rows. In Excel 2003 it would take over sixty sheets and probably some VBA code to try it. I tried the import in Excel 2007- it takes 20 seconds just to tell me I'll have to go back to the text file import multiple times to do multiple imports onto separate sheets. It took almost two minutes to do the first million rows. Even once we have the data spread across four sheets it's not clear how to summarize millions of rows in excel easily.

Instead, let's use the Datamartist tool to manage this data set and generate one thats more useful.

The first analysis we will do will be on the Vendor dimension, to determine who Acme's big vendors are, and if we can negotiate some price reductions where we have leverage.

In Datamartist, very large files are not an issue because the tool can load in only preview data- this means that it's possible to look at a sampling of a few hundred thousand rows, and design the transformation before running it on the whole data set.

The PO Detail file has the columns shown- let's answer the question - "Who are our biggest suppliers"?
So which columns do we need? We probably want to have some sense of trends over time so we'll keep the order date, but summarize to Month, we'll keep the Vendor ID of course, and then we need to use the Quantity and Price fields to calculate the total amount spent. Then we want to write this summarized data into Excel to check it out.

To do this in Datamartist all it takes is four simple blocks; A Text import block to load in the PO_Detail.txt file, a calculate block to multiply QTY by PRICE, a Summarize block to do all the summarizing, and an Excel export block to generate the excel file;

Each block passes its result to the next block via the connectors, and the last block saves it to an excel file we've specified.

Defining the calculation uses standard spreadsheet functions- here's what the config area looks like;

And defining the summary is as simple as it looks- pick the columns you want, and select what kind of summary you want done.

We run it on a preview set of 100 thousand rows (takes about twelve seconds to run), and check the output.

It looks good, so we run on the whole 4 million rows;

About seven minutes later we have our result- an excel sheet with a manageable 130 thousand rows, total spend, by vendor, by month for four years;

Next up we need to create our vendor dimension, and join it to this mini fact table we have created. Stay tuned.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Data Size Limits in Excel

There are different types of limits-

1. The size in rows and columns the actual spreadsheet has.
2. Excel's (and your PC's) ability to crunch the numbers in a reasonable time. (RAM, CPU)
3. The size of the files involved and load and save times.

In Excel 2003, a spreadsheet has rows 1 to 65 536 and columns A to IV. This makes it a grid 256 X 65536. In Excel 2007 the spreadsheet is much, much larger, with rows from 1 to 1 048 576 and columns from A to XFD. (Making a grid 16384 X 1 048 576). Now before you get too excited about how much space you have in 2007, the reality is that limits number 2 and 3 define how you can actually use that space. But it is more and more is good.

So lets kick the tires on large data sets in Excel 2007. For these very informal tests I'm using a Quad-core workstation with 4Gb of RAM, so the results I get represent a best case compared to a typical laptop or desktop PC. First of all- putting a million rows of data in Excel 2007 (even a "narrow table" of only 3-4 columns) slows everything down. Delete a column, and you'll often see a 5-10 second freeze-up while excel churns away in the background- roughly the same amount of time needed to save the file. Plus, when I push it I've had it lock up on me a few times- requiring some Ctrl-Alt-Del action to kill it. Even a narrow table such as this makes the Excel file be at minimum 15-20 megabytes. For the particular text file I used, the .txt version was 9 Mb, the .xlsx file was double the size at 18 Mb. I added a few columns and the file quickly became 80 Mb.

Also, strangely, doing exactly the same thing multiple times results in very different times to complete- when I'm mentioning times its the average of 2-3 trials (see graph).

 All in all, although Excel 2007 can technically store a million rows, I'd advise against it. There are other reasons its a pain- scroll bars and page-up page-down don't scale well to 1M rows- its just hard to copy 250000 rows accurately- takes for ever to get to the end, and then you overshoot by a mile, and page up again forever to find it etc. etc. (And yes you can use the Go To command on the Home>Editing>Find and Select>Go to - but a model of ease its not.

I can tell you, however, that using all the other features on more reasonable data sets (up to say, 100 k rows), I LOVE what it can do in terms of analysis and reporting. Once you have the data in reasonable result sets, there is no better place to have it than in Excel if you want full control in my opinion. But how to get it there. Next posts: how to link to data in Access and build a mini personal data mart. We'll learn how to make a personal data mart given the currently available tools. (And you just know there will be some posts later where I show you how to do the same thing, but using Datamartist. ) Update: Datamartist now available. Download the tool now, and find a whole new way to transform and managed your data, including managing huge data imports into excel.