In the data analysis realm, spreadsheet errors are often used to try to dissuade companies from letting their people "work with the data directly". Software vendors of all sizes, but particularly the really big ones (those incumbants) spread FUD because if they can stop people from getting at the data themselves, it increases the chance of companies buying some more business intelligence suites.

The argument goes something like this:

Spreadsheets have been shown to be plagued with errors, many studies showing error rates above 90%. You need to reduce the risk that spreadsheets are creating in your organization by establishing formal, documented processes that are created an managed by professionals using sophisticated tools.

Then the usual nightmare scenarios are brought out, all involving rabid Auditors, Sarbane-Oxley, governance failures etc.

Now, don't get me wrong, spreadsheet errors are a very real and serious problem, and there are all sorts of data applications that should never be done in Excel or other ad-hoc, user driven tools. Ever. Formal documented processes are critically important, and there are lots of places where you better be using the right tools and professionals.

I have seen the culture of the spreadsheet completely undermine initiatives that would have driven better data quality, data analysis and business processes. The spreadsheet certainly has its dark side.

But the problem is that FUD paints with a broad brush. People take it as "Spreadsheets with data in them? Bad news. Don't do it. Individuals able to get at the data, and quickly transform it, analyze it? Who knows what they'll do- shut them down!"

Sadly, from a data quality point of view, sometimes the spreadsheets have the BEST data quality- because people have fixed the issues they can't fix in the transactional system due to constraints or IT department delays.

Encourage positive change with reasonable controls.

Intelligent, responsible people should be encouraged to use "informal" methods and tools to do data analysis.

These people will find things, learn things, and drive positive change (including change in those big formal professional systems).

They should do it with a reasonable understanding that doing things in an informal way, with spreadsheets or other tools does introduce errors, and should consider this when they recommend taking action based on the results.

Balance between two extremes

The totalitarian state: I don't think there is an IT department in the world that is capable of stopping all unofficial data analysis. In fact, I would suggest that the moment such an IT department comes into existence, it would kill the host company, a harsh sort of self-regulation. People interested in data and thinking for themselves would just pack up and leave. So who would be left making the decisions and based on what?

The twisted web of spreadsheets: Companies that allow an anything goes, visual basic code, macros and manual cut and paste direct to the annual report environment are not going to be long for the world either. They populate the horror story pages on the spreadsheet risk websites.

The zone of win.

You want to be somewhere between insane spreadsheet addiction and strict formal big tool paralysis.

I submit that companies that balance risk while still encouraging their smart people to "play" with the data and do analysis in new and interesting ways with new tools are going to win.

Again, don't let this process generate your profit and loss statement- understand where and what the informal discovery process is for- but do let it discover things. If it discovers something interesting you'll have the chance to check for the errors. Make sure its part of the process to do so.

By letting the FUD get you down, you'll never get that far and who knows what insights you might be giving up?

Of course, we believe you should go even further and give those intelligent, responsible people new tools that are less error prone than spreadsheets but still provide as much or even greater flexibility. That's why we're building Datamartist after all.

Openness, balance, and clear minded pragmatism will get you further than FUD every time.

On a whim (ok, I was procrastinating at work on social media sites when I should have been doing data analysis), I decided to do a twitter search for things excel like, including "vlookup" that granddaddy of all Excel functions.

Heard recently on twitter;

I just discovered VLOOKUP in excel. This may just be the best day of my life.

I predict your life will get better and better. (It's obviously been no great shakes so far.)

lol, I'm such a banker...I haven't done this stuff in Excel in over 2 yrs, but after like 30 min, I'm back vlookup-in' my a@$ off!

Go, go go!

omg excel documents are boring!

Data is never boring, just misunderstood.

If only Excel would stop freezing, things would be better.

Amen.

Excel crashes everytime I try and make a pie chart. How am I meant to do my report now?

Use a bar chart, you should never use pie charts.

And from the overworked department:

VLOOKUP + CTRC C + CTRL V for 10 hours today.. What a life

starting to see everything in cells and formulas... =IF(ISERROR(VLOOKUP(HUNGRY,DONUTS:BURGER,1,FALSE)),"GO HOME & SLEEP","DINNER")

I hate spreadsheets and I hate data alignment, thankfully I like VLookup

My method of teaching VLOOKUP to people involves telling them to look for the banana in the fruit bowl. Strange? Yes. Effective? Absolutely.

I tend to use the watermelon analogy, but whatever works for you.

I fought Excel and...I won! VLOOKUP function, I own you! (insert evil laugh here)

so boring! you wan't to help me on my essey question? Explain how a #VLOOKUP and a #HLOOKUP benifit business? LOL

Just did an excel spreadsheet with nested VLOOKUP functions. I feel dirty.

I've showed you how to do a vlookup 5x already. If you can't figure this out, mebbe you shld look for a new job.

A job that doesn't require Excel skills? Is that possible?

And one final sentiment:

Excel should die.

There is an excel tweet about every 15 seconds. No wonder the global economy is having some issues on the rebound. Get to work people!

They have decided not to include a "Vista mode", citing lack of demand.

This is a very smart move on the part of Microsoft. Aside from the general bad press that Vista got (making users less likely to embrace a change), one of the key things that stops an IT department from recommending moving to a new operating system is the concern that certain desktop applications won't run on the new version. XP mode offers a way to ensure that legacy applications will ALWAYS run on this new version of the OS. Of course, in software "always" should perhaps never be capitalized, but running on a virtual machine its highly likely that your desktop apps that run on XP will still run on Windows 7.

Of course, the key here is not the technology side- its the fact that the XP license will be included in the Windows 7 license. You could have run a VM on windows 7- but no company will buy double the OS licenses it needs. Microsoft has just lit a rocket under Windows 7 in my opinion.

I imagine right now across the world IT departments are convening meetings to plan their Windows 7 rollout- and arguing about if they wait for service pack 1 or not.

Excel 2007 Adoption

Although good numbers are hard to come by, it seems like Excel 2007 has suffered a similar fate to Windows Vista, becoming the version for which people decide "we'll just skip this one."

Looking at a number of different sites that have surveys or discuss general estimates in terms of Excel 2007, and using a completely unscientific method of making up numbers that seem to be about the average, (see chart) we can see the early adopters, about 25% that use only Excel 2007. Then 25% percent use both (perhaps because some of their collegues are in the first 25%, so they have to be able to open those @#@*%#@ Excel 2007 files). And fully 50% are content with their previous version.

What is particularly interesting about these numbers (keeping in mind that my data sources and particularly my methods are suspect) is that they seem to be relatively stable for the last year. Regardless if the sources data was recent, six months or a year ago, it seems that the early adopters moved, and now everyone else is waiting. Waiting for the next version? Waiting for the Windows 7 upgrade that is coming?

The interesting question is "Will the next version of Excel entice them over?". We will find out in 2010. I'm going out on a limb here, but I don't think its a question of the new versions not having enough features. We're seeing a continued evolution in the business intelligence space, including desktop applications such as datamartist because existing tools just don't do what people need, but obviously spreadsheets are a more mature market. Of course Microsoft is also positioning Excel to be more of a desktop BI tool (in many ways it already is) but at the core its still a spreadsheet, and people know what they want from a spreadsheet.

Will Excel 2003 become the longest running application ever? And will every future version of Windows have the older versions supported with a virtual machine feature? Will Windows 8 have Windows 7 mode, that includes XP mode? Questionable.

Legacy software has been a reality in the enterprise world for decades, and this is officially sanctioned legacy at the operating system level. It used to be that the realities of change forced a certain number of the legacy applications to be re-written and made at least the desktop environment refresh relatively quickly.

It may be that that is about to change.

Microsoft Access has a clear role to play when a small, light database application is required. However, it has a learning curve, and is not necessarily the best tool for data analysis.

Product Segmentation Query Example

Lets look at an example ms access query or two and see how we can do the same thing Datamartist, only without the queries and without any SQL. For this example, lets say that we have two sets of sales data from different time periods, and a product list, and we want to define some product segments based on color and price. We want to get a summary of the sales Qty and average price sold by month, broken out by the new categories which are as follows:

• "Red and High Priced" If the product is Red and its minimum price is more than $1000
• "Red Low Price wide price range" If the product is Red, has a minimum price less than $1000 but has a min to max price of more than $200
• "Red Low Price small price range" If its Red and not in the first two segments
• "Yellow" if the product is yellow.
• "Other" for all the rest

The three data tables we have are as follows:

1. Sales 03-06 with about 120 000 rows, which contains sales data from 2003 - 2006
2. Sales 2007 with about 30 000 rows, which contains sales data for 2007
3. Products which contains the colors for all the products and their minimum and maximum prices

So- first step is to combine the two data tables, in Access, this is done with a UNION query with the following SQL code:

select * from [Sales Data 03-06] UNION select * from [Sales Data 2007];

In Datamartist, we simply connect the two tables up to a combine block.

Next, we need to define the segmentation- again in Access this is done with a Query, this time by nesting IIF statements to add a new column called "Product_Segment" to the resulting query.

SELECT Products.Product_ID, Products.Product_Name, Products.Product_Group, Products.Product_Category, Products.Product_SubCategory, Products.Shipping_Weight, Products.Color, Products.Price_Min, Products.Price_Max, IIf([Color]="Red" And [Price_Min]>1000,"Red and High Priced",IIf([Color]="Red" And ([Price_max]-[Price_min])>200,"Red Low Price wide price range",IIf([Color]="Red","Red Low Price small price range",IIf([Color]="Yellow","Yellow","Other")))) AS Product_Segment
FROM Products;

In Datamartist, we use a segmentation block to do the same thing. The interface is graphical, and the syntax is the same as you would use in Excel. There is no need to nest any IF statements, because the overall block is designed to do that. Heres what the blocks look like- the MS Access import block on the left, and the segmentation rule block on the right.

Each segment has the statement that defines if a row is in the segment or not. The block tests each segment rule in order, starting at the top- the first statement that solves as "TRUE" defines the value for the Product_Segment column for that row. Dragging the segments up and down changes what order the rules are checked.

(Click to Enlarge)

Then we have to Join this new product dimension (with the segmentation column) to the sales data, and summarize.

In MS Access, this is done with more queries- Heres what Access looks like when we're done.

Compare that list of Tables and Queries to the visual, left to right layout of the Datamartist data canvas that does the same thing. Without ever having to write any SQL code:

The VISUAL way to do it

In Datamartist you can see the flow of the data, the row counts are clearly displayed, and clicking on the connectors will bring up the underlying data set in the data viewer. Its clear which block feeds which, and by adding more blocks and connecting them at the desired point in the data flow, new analysis can be created.

Take Datamartist for a trial run- download it now because maybe you don't have to learn microsoft access queries after all.

There was a link posted on meta-filter today that sparked some lively discussion amongst the crowd. The Excel auto formating "feature" loves to scramble common genetic nomenclature.

It turns out that in the genetics field, common codes get converted to incorrect values regularly. One example given was the code for tumor suppressor "DEC1" which gets coverted to the date December 1. Another was the code "2310009E13" (apparently a "RIKEN clone identifier") - which would be converted to a number, 2.31E+19. I'm not a geneticist but I can just see how this wouldn't be helpful.

I checked these examples on Excel 2007, and sure enough, the default will make changes right at import that scramble the mentioned codes- no error, no notification, no problem. Of course Excel is perfectly capable of handling this data properly- the user needs to specify the field as text, and the conversions won't be done.
The key point brought up in the article (and is always true about excel spreadsheets) is not just that in this case the data gets corrupted but that depending on how carefully a user checks the error may not be detected.
If undetected, what decisions, conclusions and actions will be taken based on the incorrect information?

Excel is super powerful, and super useful, but we have to always remind ourselves to balance the ease of use with how critical our data is, and what the impact of errors might be. In the end, as with all computer use, we have to test, validate and test again at a level consistent with whatever use we are putting the data to.

Knowing this, it's not surprising that a huge amount of data collection, extraction, and transformation happens in Excel spreadsheets, or Access databases that are made without the involvement (and often under the direct scorn of) the IT department in large companies. In my previous life I was in the IT department, and I saw some amazing systems generated with hundreds of spreadsheets and databases. This mix of spreadsheets and databases, created without the involvement of the IT department by power users or external consultants (financed out of departmental budgets) are often referred to as Spreadmarts or Shadow Systems.

For an interesting survey on the subject, take a look at TDWI's report "Strategies for Managing Spreadmarts: Migrating to a Managed BI Environment". This report is now a year old, but I'm certain as valid as ever.

The title suggests that the solution is managed BI- I won't get into that right now, but you'll notice the study was sponsored by the likes of Microsoft, Cognos, Microstrategy and SAP- so of course the solution is Big Business Intelligence solutions.

But what's really interesting from the survey, is how the different groups within the respondent companies feel about spreadmarts and shadow data systems. The analysts love them, the executives are unsure, and IT hates with a passion. This makes for an interesting mix.

This is very much what I've seen in my experience. IT and the Business are at odds with each other, and senior management is either disinterested or forced to take sides.

Where do I stand? I'm in the "avoid them if you can" camp when we're talking about a tangle of spreadsheets and undocumented MS Access databases that can be error prone and time consuming. I understand why it's often unavoidable, but I've seen first hand how painful these systems are to maintain.

On the other hand, I don't subscribe to the school of thought that says "Excel needs to be eliminated- analysts should use the Business Intelligence systems only, otherwise there will be chaos." Let's not go overboard. Excel and spreadsheets are useful tools, and have their place. Additionally, I really feel for business users who simply can't get what they want from the IT departments. I used to be the IT department, and it was frustrating to not have the resources available to build what people needed.

As one of the authors of the above report, Rick Sherman, said in a recent podcast:

"reality is no matter how many IT folks that you have in your company you're not likely to have enough resources or time to meet every business users reporting or analytical requirements..."

He presents what is a refreshingly balanced approach to Excel. In his predictions for trends in 2009, number 5 is "Excel becomes an accepted tool in a BI portfolio". He points out that this may not be mainstream in 2009, but I hope he's right about the trend. A pragmatic, inclusive strategy with more power to the people while avoiding the chaotic side of spreadmarts is where the solution is.

Data Size Limits in Excel

There are different types of limits-

1. The size in rows and columns the actual spreadsheet has.
2. Excel's (and your PC's) ability to crunch the numbers in a reasonable time. (RAM, CPU)
3. The size of the files involved and load and save times.

In Excel 2003, a spreadsheet has rows 1 to 65 536 and columns A to IV. This makes it a grid 256 X 65536. In Excel 2007 the spreadsheet is much, much larger, with rows from 1 to 1 048 576 and columns from A to XFD. (Making a grid 16384 X 1 048 576). Now before you get too excited about how much space you have in 2007, the reality is that limits number 2 and 3 define how you can actually use that space. But it is more and more is good.

So lets kick the tires on large data sets in Excel 2007. For these very informal tests I'm using a Quad-core workstation with 4Gb of RAM, so the results I get represent a best case compared to a typical laptop or desktop PC. First of all- putting a million rows of data in Excel 2007 (even a "narrow table" of only 3-4 columns) slows everything down. Delete a column, and you'll often see a 5-10 second freeze-up while excel churns away in the background- roughly the same amount of time needed to save the file. Plus, when I push it I've had it lock up on me a few times- requiring some Ctrl-Alt-Del action to kill it. Even a narrow table such as this makes the Excel file be at minimum 15-20 megabytes. For the particular text file I used, the .txt version was 9 Mb, the .xlsx file was double the size at 18 Mb. I added a few columns and the file quickly became 80 Mb.

Also, strangely, doing exactly the same thing multiple times results in very different times to complete- when I'm mentioning times its the average of 2-3 trials (see graph).

 All in all, although Excel 2007 can technically store a million rows, I'd advise against it. There are other reasons its a pain- scroll bars and page-up page-down don't scale well to 1M rows- its just hard to copy 250000 rows accurately- takes for ever to get to the end, and then you overshoot by a mile, and page up again forever to find it etc. etc. (And yes you can use the Go To command on the Home>Editing>Find and Select>Go to - but a model of ease its not.

I can tell you, however, that using all the other features on more reasonable data sets (up to say, 100 k rows), I LOVE what it can do in terms of analysis and reporting. Once you have the data in reasonable result sets, there is no better place to have it than in Excel if you want full control in my opinion. But how to get it there. Next posts: how to link to data in Access and build a mini personal data mart. We'll learn how to make a personal data mart given the currently available tools. (And you just know there will be some posts later where I show you how to do the same thing, but using Datamartist. ) Update: Datamartist now available. Download the tool now, and find a whole new way to transform and managed your data, including managing huge data imports into excel.

Interestingly, there seems also to be a significant population who use both versions at once. (Its possible-  THANK YOU MICROSOFT- to install both versions at the same time.)

Being in the software development business, and since my company is a registered Microsoft Partner, I have access to lots of Microsoft software, for test and internal use purposes.  I have lots of different combinations of XP, Vista, Office 2003, Office 2007 with various options installed on my virtual machines for testing and design work.

But for my personal use? The long and the short of it is that I had Excel 2007 installed on my primary machine for about 48 hours. 

It wasn't that I didn't like it- I just didn't have a pressing need right then, and so I didn't want to spend the time and effort of learning the new interface- yet.  I will soon, as I move into the next phases and let the Datamartist tools take advantage of its new features.  But at this point I'm doing data transformation and management work- Excel is better at the presentation side- and it seems to me that that's where most of the new features in Excel 2007 are focused on.  I'm sure its awesome-  I'm working on tools for getting data in a state to be used in it first.

And I wouldn't be surprised if many of you are in the same boat.  You do all sorts of analysis in Excel, you have come to know its interface inside and out, and well, although you have nothing against the new way of doing things... its just different, is all.

And lets face it- unless there is a specific new feature some specific need drives you to, Excel 2003 is pretty powerful for what it does. 

But we can be assured that we are not alone in this resistance to change- some of the most popular add-ins for Excel 2007 provide retro-menus to avoid searching the ribbon.

So don't feel that you are behind the times- we'll all get there when it makes sense, and when there is payback- not just because its there.

What is bad data?

 Well, some of it is easy to see, some is downright impossible to find. Lets look at an easy example; a row of data where the country is "US" and the state/province is "Ontario".

 You just know both those values can't be right.  So why did the source system let it happen-  good question- when the programmers of the application tell you let me know...

So for this easy one  should we assume that the country is right, and change the state to Michigan since that's close to Ontario?  Or maybe Ohio because they both start with "O"?

The right answer is of course to go back to the source and fix the problem-  but if you've got hundreds or thousands of users, an application that can't be modified to stop this type of error at the source, and and IT department that is overworked then that is probably not an option.

Say your company does $500 million dollars of business in a year, and the Ontario, US data represents $1500 of sales- for practical analysis purposes it just doesn't really matter where it goes-  just so you don't see Ontario as a State in the report you give the CEO.

Often the solution in Excel is to just "fix it"- but if you reload the data each month, you have to go back and fix it again and again.

Or maybe you only load in the new months data, so you don't overwrite last months.  This works until some definitions change, and now all the historical data is out of sync with the new months data.  A good example of this is if you have sales regions.  If the regions are changed (new ones added, existing ones split up or merged) then the historical data you have on your machine will have to be dumped to get the sales region codes corrected for the past.  But then all your fixes have to be redone-  could be a real nightmare if you've been using it for a while. 

Another issue is that eventually the data in the source system might get fixed- and it turns out the IT departments fix wasn't the same as yours-  Ohio rather than Michigan- so you have more discrepancies to chase down.

On top of that, even though the amounts are small, its disconcerting to see different spreadsheets show different totals because you "fixed it" in one of them, and didn't in the other even just within your spreadsheets- not to mention the spreadsheets of your colleagues.  I hate it when my boss sees three different numbers.

The key is to create a single data-set from all your sources, fix the problem once, and do it in a way that the entire data set can be refreshed automatically, the fixes (those that still apply) can be "re-run" and then all your spreadsheets link to THAT single version of your truth.  The more you share this "master" sheet with your co-workers the better.

Of course, that's the whole point of having data warehouses and data marts and Enterprise business intelligence systems.  But what if the analysis you need to do hasn't been covered, or isn't scheduled to go live for another eight months?

If its just you and your best friend Excel, then here are some pointers;

• Stage your data.  Don't make 10 spreadsheets that all take data from the raw source, rather make one spreadsheet that is the "Fixed data", and have all other spreadsheets link to this. This will mean you will have a single "staging" spreadsheet, and then a number of "reporting" spreadsheets.
• Record all your "fixes" on a sheet in this staging spreadsheet called "Known Issues".  The ideal would be to automate it so that the fixes get applied each time you reload the entire data-set, but by at least having a clear record you wil be able to quickly get the data where you want it each time you reload.
• Don't think about report layout or formatting in your staging spreadsheet-  keep the data in simple tables that is more aligned with how you get the raw data- every column has the same values in it all the way down.  If you have different data-sets, use different sheets.  Do the reporting in your reporting spreadsheets where you can have tables with different mixes of data from different sources.

The more macros or scripts you can use the better- but even if you have a set order of cut and paste, and follow it to the letter (write it down on a sheet called "Refresh Steps" in the staging spreadsheet maybe) it will reduce the amount of time it takes to update each time new data is available, and you only do the fixes once, and if you've linked it cleanly to the reporting sheets then the rework will be reduced.

 Of course, its still a lot of work.  In a nutshell that's why I'm working on the Datamartist tool that will automate this and much more, allowing you to easily and without programming be able to manage your spreadsheets much more effectively.  Stay tuned, and in the meantime happy staging.