Microsoft Access has a clear role to play when a small, light database application is required. However, it has a learning curve, and is not necessarily the best tool for data analysis.

Product Segmentation Query Example

Lets look at an example ms access query or two and see how we can do the same thing Datamartist, only without the queries and without any SQL. For this example, lets say that we have two sets of sales data from different time periods, and a product list, and we want to define some product segments based on color and price. We want to get a summary of the sales Qty and average price sold by month, broken out by the new categories which are as follows:

• "Red and High Priced" If the product is Red and its minimum price is more than $1000
• "Red Low Price wide price range" If the product is Red, has a minimum price less than $1000 but has a min to max price of more than $200
• "Red Low Price small price range" If its Red and not in the first two segments
• "Yellow" if the product is yellow.
• "Other" for all the rest

The three data tables we have are as follows:

1. Sales 03-06 with about 120 000 rows, which contains sales data from 2003 - 2006
2. Sales 2007 with about 30 000 rows, which contains sales data for 2007
3. Products which contains the colors for all the products and their minimum and maximum prices

So- first step is to combine the two data tables, in Access, this is done with a UNION query with the following SQL code:

select * from [Sales Data 03-06] UNION select * from [Sales Data 2007];

In Datamartist, we simply connect the two tables up to a combine block.

Next, we need to define the segmentation- again in Access this is done with a Query, this time by nesting IIF statements to add a new column called "Product_Segment" to the resulting query.

SELECT Products.Product_ID, Products.Product_Name, Products.Product_Group, Products.Product_Category, Products.Product_SubCategory, Products.Shipping_Weight, Products.Color, Products.Price_Min, Products.Price_Max, IIf([Color]="Red" And [Price_Min]>1000,"Red and High Priced",IIf([Color]="Red" And ([Price_max]-[Price_min])>200,"Red Low Price wide price range",IIf([Color]="Red","Red Low Price small price range",IIf([Color]="Yellow","Yellow","Other")))) AS Product_Segment
FROM Products;

In Datamartist, we use a segmentation block to do the same thing. The interface is graphical, and the syntax is the same as you would use in Excel. There is no need to nest any IF statements, because the overall block is designed to do that. Heres what the blocks look like- the MS Access import block on the left, and the segmentation rule block on the right.

Each segment has the statement that defines if a row is in the segment or not. The block tests each segment rule in order, starting at the top- the first statement that solves as "TRUE" defines the value for the Product_Segment column for that row. Dragging the segments up and down changes what order the rules are checked.

(Click to Enlarge)

Then we have to Join this new product dimension (with the segmentation column) to the sales data, and summarize.

In MS Access, this is done with more queries- Heres what Access looks like when we're done.

Compare that list of Tables and Queries to the visual, left to right layout of the Datamartist data canvas that does the same thing. Without ever having to write any SQL code:

The VISUAL way to do it

In Datamartist you can see the flow of the data, the row counts are clearly displayed, and clicking on the connectors will bring up the underlying data set in the data viewer. Its clear which block feeds which, and by adding more blocks and connecting them at the desired point in the data flow, new analysis can be created.

Take Datamartist for a trial run- download it now because maybe you don't have to learn microsoft access queries after all.

Let's face it, if you're considering Microsoft Access chances are what you need to get done is beyond what Excel does well, so you're looking for options. Its also likely that your IT department is unable or un-willing to help you out- this being even more likely as the recession reduces reporting budgets left, right and center.

Two of the key things that lead someone to search for a database solution are:

• Data Volume- More than a million rows and Excel becomes very difficult, even before that the performance suffers.
• Flexibility to Join Tables - Vlookup and VBA code only go so far- Access gives an easy way to make joins between tables, one of the powerful features of relational databases.

Now, the data volume is what it is- if you have millions and millions of rows, you need something to cut it down to size before you move it into your Excel spreadsheet.

On the other point, however, I can hear the Excel fans saying "now wait a minute, Excel can do that, I don't really need a database" and they are right. But they are almost always right- Excel can do almost anything. It does not mean, however that its the best tool for the job. Using Vlookup and VBA scripts to join up multiple tables is not my idea of a fun time. And even in Excel 2007 I find the pivot tables annoying and prone to break if I'm adding categories, moving data sets or heaven forbid changing number and order of columns.

Microsoft Access has a very nice interface for creating joins between tables, just a simple drag and drop between fields. The cross tab query capability is useful and good, and being a relational database it's more tolerant of changes to table structure because it's not messing with cell references.

"But", many who have used MS Access will say, "its pretty complex to learn, and even if I do start to get the query stuff down, it doesn't handle bad data well."

Bad data? Who has bad data? Isn't all data pristine, as intended, correctly formatted and accurate?

One of the huge differences between Excel and MS Access is that Excel is extremely flexible. (Probably more flexible than your auditor would like, but thats a different story). One source of Excels flexibility is its ability to accept different data types in the same column, and to allow editing of cells quickly. In Microsoft Access, for example, when it sees some variation it either discards the data or defaults to the data type "Text"- meaning now you can't perform the calculations you need to do on your data.

This illustrates one of the challenges people face in trying to use a database - databases are very strict on data types. Once you declare a data type for a column, if you import data into the table, the database will discard the values that do not conform to that data type. In Excel, you get cell errors if you try calculations but the original data is still there.

One of the powerful features of the Datamartist tool is the fact that it has an underlying database structure that provides flexibility on data types. Unlike MS Access and other databases, Datamartist can store dates, numbers, strings and booleans natively in a single column. (It does not convert to strings- it stores the full object). Take a look at this example:

In each individual row, Datamartist completes the calculation if possible. Datamartist is a database that gives you the freedom of a Spreadsheet. Of course, just like excel, if you ask for a calculation on a value that is meaningless you will get an error- but at the individual value- not a full row discard. This means that with messy data you can still work with it, bring it in, and fix it. In Access or another database, you can't even get it through the front door (or it defaults to text, making many calculations impossible).

This won't be the last time I compare these three tools- and the types of data structures and tasks each of them are most effective with.

In the mean time- Download Datamartist- see what I'm talking about with your own data.

Knowing this, it's not surprising that a huge amount of data collection, extraction, and transformation happens in Excel spreadsheets, or Access databases that are made without the involvement (and often under the direct scorn of) the IT department in large companies. In my previous life I was in the IT department, and I saw some amazing systems generated with hundreds of spreadsheets and databases. This mix of spreadsheets and databases, created without the involvement of the IT department by power users or external consultants (financed out of departmental budgets) are often referred to as Spreadmarts or Shadow Systems.

For an interesting survey on the subject, take a look at TDWI's report "Strategies for Managing Spreadmarts: Migrating to a Managed BI Environment". This report is now a year old, but I'm certain as valid as ever.

The title suggests that the solution is managed BI- I won't get into that right now, but you'll notice the study was sponsored by the likes of Microsoft, Cognos, Microstrategy and SAP- so of course the solution is Big Business Intelligence solutions.

But what's really interesting from the survey, is how the different groups within the respondent companies feel about spreadmarts and shadow data systems. The analysts love them, the executives are unsure, and IT hates with a passion. This makes for an interesting mix.

This is very much what I've seen in my experience. IT and the Business are at odds with each other, and senior management is either disinterested or forced to take sides.

Where do I stand? I'm in the "avoid them if you can" camp when we're talking about a tangle of spreadsheets and undocumented MS Access databases that can be error prone and time consuming. I understand why it's often unavoidable, but I've seen first hand how painful these systems are to maintain.

On the other hand, I don't subscribe to the school of thought that says "Excel needs to be eliminated- analysts should use the Business Intelligence systems only, otherwise there will be chaos." Let's not go overboard. Excel and spreadsheets are useful tools, and have their place. Additionally, I really feel for business users who simply can't get what they want from the IT departments. I used to be the IT department, and it was frustrating to not have the resources available to build what people needed.

As one of the authors of the above report, Rick Sherman, said in a recent podcast:

"reality is no matter how many IT folks that you have in your company you're not likely to have enough resources or time to meet every business users reporting or analytical requirements..."

He presents what is a refreshingly balanced approach to Excel. In his predictions for trends in 2009, number 5 is "Excel becomes an accepted tool in a BI portfolio". He points out that this may not be mainstream in 2009, but I hope he's right about the trend. A pragmatic, inclusive strategy with more power to the people while avoiding the chaotic side of spreadmarts is where the solution is.

What is our goal? To build a data mart.  What is a data mart?  A data mart is data that has been formatted for ease of analysis, and that contains the information that an analyst needs, even if that information was not in the original system or at least not in a format that is easy to use.  The best way to model a data mart is to build it using two types of tables.

1. Data-  the FACTS - which define the who, what where, when of the data.
2. Definitions - the DIMENSIONS - which describe the various things that are found in the Facts.

An example would be in order.  Lets say we have some sales data that we are analysing.  Ideally, we want to get this information in a format where the FACTS are that on date A, we sold product Y to customer Z with a quantity and a price.  So in MS Access the columns might be:

Note that if we can we use Unique things to define the who what where when- for example, we don't use the Customers name (how many John Smiths are there?) but we use a unique customer number or ID.  Then we need three definition sets- Date, product and customer that match these IDs.  By the way- creating these IDs when they don't exist, and cleaning the duplicates up is one of the challenges- if you already have a clean set of customer numbers and data then you are way ahead.

This is the key in terms of the data model- Sales_FACT holds JUST THE FACTS, and a table for each definition or DIMENSION, that holds just once the definition for each unique Product or customer.  This way we have consistent information throughout the analysis- if something about a product changes we don't have to change it in every row of the FACTS, we only have to change the single entry in, say, the product DIMENSION. The Product and Customer dimension tables might look like this (again in Access):

The final step in analysis is to combine the FACT and the DIMENSION to form a table or view that has all the information you need to slice and dice- often using a Pivot table in Excel is a good way to do this, or simply autofilters etc.  But first in access it is necessary to create a query.  This can be done by selecting "create" on the ribbon control, then design view, and showing the three tables we've created- the Sales_Fact, Product_DIMENSION and Customer_DIMENSION.  Now its necessary to JOIN these tables together using the ID fields- this is done by dragging the Product_ID field from the Product_DIMENSION table and dropping it on the Product_ID field on the Sales_FACT table, and then dragging the Customer_ID field from the Customer_DIMENSION table and dropping it on the Customer_ID field on the Sales_FACT table.

 This results in a query that returns all the rows in the table Sales_FACT with the product and customer information looked up from the two dimension tables and added in.

If anyone cares- the Structured Query Language (SQL) that is generated by Access was as follows:

SELECT Sales_FACT.[Sales Date], Sales_FACT.Quantity, Sales_FACT.Price, Product_DIMENSION.Product_Name, Product_DIMENSION.Product_Category, Customer_DIMENSION.Cust_Name, Customer_DIMENSION.Cust_Email, Sales_FACT.Product_ID, Sales_FACT.Customer_ID
FROM Customer_DIMENSION INNER JOIN (Product_DIMENSION INNER JOIN Sales_FACT ON Product_DIMENSION.Product_ID = Sales_FACT.Product_ID) ON Customer_DIMENSION.Customer_ID = Sales_FACT.Customer_ID;

This table structure is a very simple example of a STAR SCHEMA- the basic data model used by respectable data marts and data warehouses everywhere.

This is all good and well, if you happen to have the data in the right format- but usually the data is stored (or created) in the source system using very different data models than this simple star schema.  This is done to allow the source system do transactions (both reading and writing data) quickly.  The star schema is excellent for analysing data, but not effective for modifying it. 

In the end this is the key as to why we need data transformation tools.  Data in the source systems is often in the wrong format for analysis, or has quality issues that while perhaps not ideal, do not actually cause the transactional system to fail, so may not get addressed, but will radically affect your analysis.  The transformation tool resolves these issues, and separates the data into the Fact, Dimension structure.  This can be done with access, although if there are multiple transformations fairly complex queries or scripts must be created.  Stay tuned to this channel- Datamartist will show you some tricks without having to figure through all the INNER JOIN SQL statements that tools like Microsoft Access needs to use...