I believe a key to understanding how information moves around your organization is to think of it as a mini economy. (I know, the economy is not our favorite subject right now, but bear with me).

There are information suppliers, and information consumers. The consumers are willing to pay more or less for different types of information, and different methods of supplying information have different costs. In the end, the market decides what gets done and what does not get done.

And like many markets, there is also a underground economy- places consumers go if the official prices don't make sense, or the products they want are not available on the open market.

In many companies, the IT department in theory has a monopoly on information supply, however the underground is active and constitutes a significant supply. The underground in this case is all the excel spreadsheets, the MS Access databases etc. used to make the shadow systems and spread marts. Spreadmarts seem to exist in the majority of enterprises- I've mentioned an interesting study regarding these shadow systems previously, and the attitudes people have.

To help illustrate this I am going to make up some data and put it in colorful graphs.

Looking at the first graph, in broad terms a data warehouse based approach will have higher costs than one based on data marts (because data warehouse provide more cross enterprise integration, which requires more effort), and the spreadmarts will have the lowest perceived cost. It's important to note that the actual cost of spreadmarts are higher, but percieved cost is what drives the consumers choice.

The trick is that because the percieved cost of spreadmarts is so low, and because there is no sanctioned enterprise solution to compete, a significant amount of effort is put in to these systems for any type of analysis that is percieved to be possible. Of course for certain data volumes or complexities there is no alternative to a full fledged data warehouse or data mart project, but for almost everything else, business users and analysts will often try to go it alone creating a chaos of spreadsheets and data bases.

The problem is, even "experts" can't accurately estimate how much effort the data analysis is. So estimates for how long it will take to "whip it up in excel" by non-experts are almost always low by orders of magnitude.

Don't dictate. Engage with sanctioned tools that work the way people want to work.

The key to adjusting this market imbalance is to introduce a new sanctioned product line, in effect undercutting the "black market".

This is exactly what self serve data transformation is about. Rather than leaving users to do it themselves in Excel- IT can provide specific tools, and thereby reduce the amount of completely opaque data transformation going on, while still providing users with the ability to get what they need.

So why is that better?

• It opens up the dialog - Talking is better than having a "Us" vs "Them" mentality. It lets you meet the people involved, lets you discuss their challenges with them, and provides an opening for discussion of important topics like data quality, master data management and data security.
• You'll know who the power users are - Right now, it is potentially anyone who has Excel- chances are that's everyone in your organisation.
• It gives you visibility on what matters to the business - If you know what the hot topics are, it can help you keep the official systems relevant and prioritize your efforts where they will do the most good.

What has to be different in this new relationship, however, is that IT has to understand about the "self" in self-serve. People will do things that no self-respecting ETL developer or data warehouse architect would ever sanction. If you clamp down and stop them, they will abandon the tools and return to the wild west. IT believes that it has the power in the relationship, but in fact the users are able to walk at any time. So add value, communicate, educate, but don't dictate. If your relationship with the business users, and the "Kings of the spreadmart" is poor to start, you have to give it time to evolve.

"But we just can't let them do that."

Resist the urge to clamp down.

Keep your systems secure, guard your infrastructure, but don't have any illusions that you can stop people from analyzing and transforming their data.

If they want to calculate net sales in a particular way then they'll do it in excel, and it will be the number that the CEO sees. The business is made up of grownups, after all. IT has a responsibility to explain the issues and challenges that shadow systems and rampant spreadsheeting can cause, but I have yet to see or hear of a company where an authoritarian approach works. As Princess Leia said- "The more you tighten your grip, Tarkin, the more star systems will slip through your fingers."

Arming the rebels

The business intelligence vendors are all realizing what the crowd pleasers are- really good integration into office applications, excel at the forefront. People want at their data.

Microsoft has of course long provided the main weapons for the shadow systems, MS Excel and MS Access- and they are going nuclear with the addition of "Power Pivot" to Excel 2010- although it is largely a presentation layer tool, and probably won't be used widely for data transformation itself.

Trying to fight all this with the standard tools of closing down the ability to export data, hiring an army of report writers, and constantly raving about the dangers and pitfalls of run away spreadsheets is like pushing on a rope.

Provide a safe, legal alternative to the free for all.

Talk to your business users. Understand their needs. Provide them with tools. Work with them to both empower responsible analysts, and avoid the worst issues that existing shadow systems are creating.

Microsoft Access has a clear role to play when a small, light database application is required. However, it has a learning curve, and is not necessarily the best tool for data analysis.

Product Segmentation Query Example

Lets look at an example ms access query or two and see how we can do the same thing Datamartist, only without the queries and without any SQL. For this example, lets say that we have two sets of sales data from different time periods, and a product list, and we want to define some product segments based on color and price. We want to get a summary of the sales Qty and average price sold by month, broken out by the new categories which are as follows:

• "Red and High Priced" If the product is Red and its minimum price is more than $1000
• "Red Low Price wide price range" If the product is Red, has a minimum price less than $1000 but has a min to max price of more than $200
• "Red Low Price small price range" If its Red and not in the first two segments
• "Yellow" if the product is yellow.
• "Other" for all the rest

The three data tables we have are as follows:

1. Sales 03-06 with about 120 000 rows, which contains sales data from 2003 - 2006
2. Sales 2007 with about 30 000 rows, which contains sales data for 2007
3. Products which contains the colors for all the products and their minimum and maximum prices

So- first step is to combine the two data tables, in Access, this is done with a UNION query with the following SQL code:

select * from [Sales Data 03-06] UNION select * from [Sales Data 2007];

In Datamartist, we simply connect the two tables up to a combine block.

Next, we need to define the segmentation- again in Access this is done with a Query, this time by nesting IIF statements to add a new column called "Product_Segment" to the resulting query.

SELECT Products.Product_ID, Products.Product_Name, Products.Product_Group, Products.Product_Category, Products.Product_SubCategory, Products.Shipping_Weight, Products.Color, Products.Price_Min, Products.Price_Max, IIf([Color]="Red" And [Price_Min]>1000,"Red and High Priced",IIf([Color]="Red" And ([Price_max]-[Price_min])>200,"Red Low Price wide price range",IIf([Color]="Red","Red Low Price small price range",IIf([Color]="Yellow","Yellow","Other")))) AS Product_Segment
FROM Products;

In Datamartist, we use a segmentation block to do the same thing. The interface is graphical, and the syntax is the same as you would use in Excel. There is no need to nest any IF statements, because the overall block is designed to do that. Heres what the blocks look like- the MS Access import block on the left, and the segmentation rule block on the right.

Each segment has the statement that defines if a row is in the segment or not. The block tests each segment rule in order, starting at the top- the first statement that solves as "TRUE" defines the value for the Product_Segment column for that row. Dragging the segments up and down changes what order the rules are checked.

(Click to Enlarge)

Then we have to Join this new product dimension (with the segmentation column) to the sales data, and summarize.

In MS Access, this is done with more queries- Heres what Access looks like when we're done.

Compare that list of Tables and Queries to the visual, left to right layout of the Datamartist data canvas that does the same thing. Without ever having to write any SQL code:

The VISUAL way to do it

In Datamartist you can see the flow of the data, the row counts are clearly displayed, and clicking on the connectors will bring up the underlying data set in the data viewer. Its clear which block feeds which, and by adding more blocks and connecting them at the desired point in the data flow, new analysis can be created.

Take Datamartist for a trial run- download it now because maybe you don't have to learn microsoft access queries after all.

With data marts (just like leasing a car) you get behind the wheel quickly, and it gets you where you want to go in style.  And the monthly payment is something you can afford now.  However, long term, well, in three years you don't own it, and have paid a bundle.

But let's be realistic.  Just as having all the cash on hand to buy the car outright just might not be in the cards,  a true data warehouse might require a very significant outlay before anything comes out the other end, making it unaffordable.  A quick, focused departmental data mart could be delivering value in a matter of weeks with relatively little investment.  (Your actual mileage may vary- depending on where you're at, its always dangerous to believe someone when they say "a matter of weeks" when software and people are involved.)

Will that departmental data mart, or even a number of data marts lead you to a single version of the truth?  Will it give you deep competitive advantage through a culture of data analytics and cross enterprise master data management? In my honest opinion, No.

But is it something you can afford in today's economy, and will you learn things about your data, your company's information culture, and your business that will be useful if in the future you embark on a true data warehouse initiative.  Yes.  Yes it is, and yes you will.

And I'll take it one (blatantly promotional) step further.  Is a personal data mart on your desk top as good as a full fledged departmental data mart with an army of highly paid developers maintaining it?  Probably not.

Is the personal data mart on your desk basicly free in comparision to the servers, software and hired help the data mart requires?- Yes. And does it, just like the data mart does for the data warehouse, prepare the ground for the next evolution when the economy turns around? Yes. Yes it does.

In difficult times companies that are pragmatic, and do what is possible, preparing for the day when more will be, survive to see that day.

It seems obvious that doing nothing because you can't afford to do the best thing is a bad strategy- but we need to ask ourselves, how often do we make that exact choice through inaction?

Why do so many people turn to Excel for their analysis needs? Because with excel they are in control.  With Excel it IS personal. Massive, expensive data warehouses are one size fits none. Excel is the ultimate in customisation. 

“Despite years of investing in BI, many IT organisations have difficulty connecting BI with the business, and to get business users fully involved and out of the ‘Excel culture’”
- Bill Hostmann, Vice President and distinguished analyst at Gartner

Excel isn't always enough

Many users frustrated by the limitations in Excel in terms of data volume and functionality want more.  Some of them take up programming macros- others spend money they'd rather not on throw away database development.  But demand encourages supply, and new tools are beginning to emerge.

As Ted Cuzzillo put it so well in his trends for 2009 blog post recently:

“on business desktops, new, highly individualised tools will sprout.”

The Datamartist vision is that of a user driven tool that provides the flexibility of a spreadsheet with the key functionality of a database without the database. 

Personal data marts can for the first time be built directly on the desktop, without programming, yet handling millions of rows of data, and implementing truly useful data analysis.