On a whim (ok, I was procrastinating at work on social media sites when I should have been doing data analysis), I decided to do a twitter search for things excel like, including "vlookup" that granddaddy of all Excel functions.

Heard recently on twitter;

I just discovered VLOOKUP in excel. This may just be the best day of my life.

I predict your life will get better and better. (It's obviously been no great shakes so far.)

lol, I'm such a banker...I haven't done this stuff in Excel in over 2 yrs, but after like 30 min, I'm back vlookup-in' my a@$ off!

Go, go go!

omg excel documents are boring!

Data is never boring, just misunderstood.

If only Excel would stop freezing, things would be better.

Amen.

Excel crashes everytime I try and make a pie chart. How am I meant to do my report now?

Use a bar chart, you should never use pie charts.

And from the overworked department:

VLOOKUP + CTRC C + CTRL V for 10 hours today.. What a life

starting to see everything in cells and formulas... =IF(ISERROR(VLOOKUP(HUNGRY,DONUTS:BURGER,1,FALSE)),"GO HOME & SLEEP","DINNER")

I hate spreadsheets and I hate data alignment, thankfully I like VLookup

My method of teaching VLOOKUP to people involves telling them to look for the banana in the fruit bowl. Strange? Yes. Effective? Absolutely.

I tend to use the watermelon analogy, but whatever works for you.

I fought Excel and...I won! VLOOKUP function, I own you! (insert evil laugh here)

so boring! you wan't to help me on my essey question? Explain how a #VLOOKUP and a #HLOOKUP benifit business? LOL

Just did an excel spreadsheet with nested VLOOKUP functions. I feel dirty.

I've showed you how to do a vlookup 5x already. If you can't figure this out, mebbe you shld look for a new job.

A job that doesn't require Excel skills? Is that possible?

And one final sentiment:

Excel should die.

There is an excel tweet about every 15 seconds. No wonder the global economy is having some issues on the rebound. Get to work people!

They have decided not to include a "Vista mode", citing lack of demand.

This is a very smart move on the part of Microsoft. Aside from the general bad press that Vista got (making users less likely to embrace a change), one of the key things that stops an IT department from recommending moving to a new operating system is the concern that certain desktop applications won't run on the new version. XP mode offers a way to ensure that legacy applications will ALWAYS run on this new version of the OS. Of course, in software "always" should perhaps never be capitalized, but running on a virtual machine its highly likely that your desktop apps that run on XP will still run on Windows 7.

Of course, the key here is not the technology side- its the fact that the XP license will be included in the Windows 7 license. You could have run a VM on windows 7- but no company will buy double the OS licenses it needs. Microsoft has just lit a rocket under Windows 7 in my opinion.

I imagine right now across the world IT departments are convening meetings to plan their Windows 7 rollout- and arguing about if they wait for service pack 1 or not.

Excel 2007 Adoption

Although good numbers are hard to come by, it seems like Excel 2007 has suffered a similar fate to Windows Vista, becoming the version for which people decide "we'll just skip this one."

Looking at a number of different sites that have surveys or discuss general estimates in terms of Excel 2007, and using a completely unscientific method of making up numbers that seem to be about the average, (see chart) we can see the early adopters, about 25% that use only Excel 2007. Then 25% percent use both (perhaps because some of their collegues are in the first 25%, so they have to be able to open those @#@*%#@ Excel 2007 files). And fully 50% are content with their previous version.

What is particularly interesting about these numbers (keeping in mind that my data sources and particularly my methods are suspect) is that they seem to be relatively stable for the last year. Regardless if the sources data was recent, six months or a year ago, it seems that the early adopters moved, and now everyone else is waiting. Waiting for the next version? Waiting for the Windows 7 upgrade that is coming?

The interesting question is "Will the next version of Excel entice them over?". We will find out in 2010. I'm going out on a limb here, but I don't think its a question of the new versions not having enough features. We're seeing a continued evolution in the business intelligence space, including desktop applications such as datamartist because existing tools just don't do what people need, but obviously spreadsheets are a more mature market. Of course Microsoft is also positioning Excel to be more of a desktop BI tool (in many ways it already is) but at the core its still a spreadsheet, and people know what they want from a spreadsheet.

Will Excel 2003 become the longest running application ever? And will every future version of Windows have the older versions supported with a virtual machine feature? Will Windows 8 have Windows 7 mode, that includes XP mode? Questionable.

Legacy software has been a reality in the enterprise world for decades, and this is officially sanctioned legacy at the operating system level. It used to be that the realities of change forced a certain number of the legacy applications to be re-written and made at least the desktop environment refresh relatively quickly.

It may be that that is about to change.

Microsoft Access has a clear role to play when a small, light database application is required. However, it has a learning curve, and is not necessarily the best tool for data analysis.

Product Segmentation Query Example

Lets look at an example ms access query or two and see how we can do the same thing Datamartist, only without the queries and without any SQL. For this example, lets say that we have two sets of sales data from different time periods, and a product list, and we want to define some product segments based on color and price. We want to get a summary of the sales Qty and average price sold by month, broken out by the new categories which are as follows:

• "Red and High Priced" If the product is Red and its minimum price is more than $1000
• "Red Low Price wide price range" If the product is Red, has a minimum price less than $1000 but has a min to max price of more than $200
• "Red Low Price small price range" If its Red and not in the first two segments
• "Yellow" if the product is yellow.
• "Other" for all the rest

The three data tables we have are as follows:

1. Sales 03-06 with about 120 000 rows, which contains sales data from 2003 - 2006
2. Sales 2007 with about 30 000 rows, which contains sales data for 2007
3. Products which contains the colors for all the products and their minimum and maximum prices

So- first step is to combine the two data tables, in Access, this is done with a UNION query with the following SQL code:

select * from [Sales Data 03-06] UNION select * from [Sales Data 2007];

In Datamartist, we simply connect the two tables up to a combine block.

Next, we need to define the segmentation- again in Access this is done with a Query, this time by nesting IIF statements to add a new column called "Product_Segment" to the resulting query.

SELECT Products.Product_ID, Products.Product_Name, Products.Product_Group, Products.Product_Category, Products.Product_SubCategory, Products.Shipping_Weight, Products.Color, Products.Price_Min, Products.Price_Max, IIf([Color]="Red" And [Price_Min]>1000,"Red and High Priced",IIf([Color]="Red" And ([Price_max]-[Price_min])>200,"Red Low Price wide price range",IIf([Color]="Red","Red Low Price small price range",IIf([Color]="Yellow","Yellow","Other")))) AS Product_Segment
FROM Products;

In Datamartist, we use a segmentation block to do the same thing. The interface is graphical, and the syntax is the same as you would use in Excel. There is no need to nest any IF statements, because the overall block is designed to do that. Heres what the blocks look like- the MS Access import block on the left, and the segmentation rule block on the right.

Each segment has the statement that defines if a row is in the segment or not. The block tests each segment rule in order, starting at the top- the first statement that solves as "TRUE" defines the value for the Product_Segment column for that row. Dragging the segments up and down changes what order the rules are checked.

(Click to Enlarge)

Then we have to Join this new product dimension (with the segmentation column) to the sales data, and summarize.

In MS Access, this is done with more queries- Heres what Access looks like when we're done.

Compare that list of Tables and Queries to the visual, left to right layout of the Datamartist data canvas that does the same thing. Without ever having to write any SQL code:

The VISUAL way to do it

In Datamartist you can see the flow of the data, the row counts are clearly displayed, and clicking on the connectors will bring up the underlying data set in the data viewer. Its clear which block feeds which, and by adding more blocks and connecting them at the desired point in the data flow, new analysis can be created.

Take Datamartist for a trial run- download it now because maybe you don't have to learn microsoft access queries after all.

There was a link posted on meta-filter today that sparked some lively discussion amongst the crowd. The Excel auto formating "feature" loves to scramble common genetic nomenclature.

It turns out that in the genetics field, common codes get converted to incorrect values regularly. One example given was the code for tumor suppressor "DEC1" which gets coverted to the date December 1. Another was the code "2310009E13" (apparently a "RIKEN clone identifier") - which would be converted to a number, 2.31E+19. I'm not a geneticist but I can just see how this wouldn't be helpful.

I checked these examples on Excel 2007, and sure enough, the default will make changes right at import that scramble the mentioned codes- no error, no notification, no problem. Of course Excel is perfectly capable of handling this data properly- the user needs to specify the field as text, and the conversions won't be done.
The key point brought up in the article (and is always true about excel spreadsheets) is not just that in this case the data gets corrupted but that depending on how carefully a user checks the error may not be detected.
If undetected, what decisions, conclusions and actions will be taken based on the incorrect information?

Excel is super powerful, and super useful, but we have to always remind ourselves to balance the ease of use with how critical our data is, and what the impact of errors might be. In the end, as with all computer use, we have to test, validate and test again at a level consistent with whatever use we are putting the data to.