Now, of course, standards are needed, and for a broad range of tools its counter productive (and horrifically expensive) to let everyone just use what they want. The cost of data centers, integration, etc. will be radically more if a company does not bring order to the chaos and the marginal advantages that a very specific or niche technology might have in one department is often obliterated by the increased support costs and integration issues globally.

But if a company looks at these savings that come from standardization, and extrapolates too far, they can fall off the other side of the benefits curve and find that they're hurting, not helping.

In the article in Harvard business review, the researchers also throw around the term "bitsmith" to describe someone who has both subject knowledge, and the ability to wrangle software, and to even create GASP! custom software that does what the team needs to get done. In many companies, current information technology dogma does not leave much room for people that have the time to be a "bitsmith".

In many companies "Custom software" is a four letter word. Well, I personally have used it, commissioned it and written it and know that often it can provide fantastic value- I've also seen people spend thousands of hours building something that never quite worked when an off the shelf tool a hundred times as good could be bought for a few thousand dollars. It's a matter of being realistic and learning how to see the difference between a problem that is specific to your industry/situation that could really benefit from some custom code, compared to a problem that is huge, main stream, and solved hundreds of times over by existing software vendors.

What I think can happen is a pendulum swing, where a company goes from "no standards, a jungle of wasteful custom software" to "Thou shalt use/buy only software on the following list."

The problem is, making a list that contains everything is just not possible. Things change. Stuff happens. It is possible that a single person might need a single piece of software that allows them to understand something, design something, communicate something that will make that software have a truly massive payback, justifying all sorts of pain and config on the part of technical resources and infrastructure.

The challenge, as always, is to have an open, working relationship between those entrusted with establishing and enforcing standards in terms of tools and those who are expected to use those tools to do business. As with so many things in governance, it's about balance, clear goals, and processes that allow for brilliance, change and creativity while not letting that process become the loop-hole that undermines all those savings standardization brings.

This is what I got when I searched for "business intelligence".

Apparently a lot of tweets containing "business intelligence" are microsoft focused. Can you even see the tiny "cognos", the "sap" in there? Oracle is readable- but wow. Microsoft. Sharepoint. Somebody in the Microsoft marketing camp needs to save this picture for when its time to talk about performance bonuses.

I've written here before about the Microsoft marketing machine, and when I did, I noticed a flurry of traffic to the Datamartist site from Redmond, and then noticed that I got some hits from twitter itself- Microsoft BI folks exchanging the link to the blog post. So the people selling and promoting Microsoft Business intelligence are definitely on twitter.

Whats also interesting, is to see the current marketing storms- guess what comes up when you search for "Microsoft", and guess what for "Google"? Yes, if you guessed "BING" for Microsoft and "Wave" for google you're right- but notice how "BING" made a double appearance- People talking about google are talking about bing, but people talking about wave aren't talking about Microsoft. I will leave it to the marketing experts to put forward their theories on all this.

Here's the secret- If you answer yes to all of the following its probably a good idea to buy:

• You've configured your Enterprise Resource Planning (ERP) software exactly as the vendor who wrote the data warehouse or data mart expected you to.
• You use ALL the modules of the ERP- and do not have any "best of breed" applications for any areas.
• You are interested in the key performance indicators that they have included and they calculate them the same way your company does
• Your end users like the layout, formating and color choices in the reports included and have promised not to ask for changes.

In fact if all this is true, then you will probably save HUGE amounts of money by using a pre-built package.

But. Lots of things tend to get in the way of that dream. More often then not it's not the pre-built package that needs to be focused on- it's what you have in your system and how you've configured it.

ERP Customization

Anyone who has been involved in a large ERP project knows that there is always at least a bit of customization. If the ERP is a good fit to your industry, and the project is focused on avoiding customization and using all those "flex fields" and "category codes" as they were intended, then the pre-built data warehouse that is available for that ERP might be a very cost effective route.

But here's the rub; in many ERP projects rather than change the existing processes (which by the way is often the justification for installing an ERP- process improvement), the business users find reasons to "keep things as they are"- and as a result, the ERP bends to the existing process rather than the other way round.

This means that new tables are created to hold information that normally would be in a different format in one of the standard ERP tables. This means that that code in field "X" that normally would mean something is overloaded to mean three or four things. This means that field "Y" that isn't needed by your company ends up being used for something completely different.

Well, when the pre-built data mart plugs into your ERP, it doesn't know anything about those tables you've added, and its assuming that the data is in the standard tables. Its already configured its ETLs, and star schemas to use those two fields that you've high-jacked, and so all those reports and cubes don't make any sense without modifications.

To be fair, many of the pre-built data warehouse frameworks have evolved to have lots of flexibility and configurability in them- you can modify and map the ETL jobs provided to fit your data. You can turn off KPIs so that they don't load in false data.

But the trick is the more you have to modify, map and configure, the more effort you spend. The more you shut off, the less you have. In the end you can find that you are always trying to fit your specific requirements into the general, vanilla requirements that came out of the box. Often, this configuration is more effort and demands more compromises than just building what you need.

Non ERP transactional systems

In some ERP implementations, certain modules of the ERP are not used and instead a third party software provides the functionality. Often, key information is then moved in and out of the ERP as required for transactions via an enterprise application integration (EAI) tool. In many cases, the reason the additional system is used is because it has more functionality (and therefore stores more detailed and varied information) than the ERP module. As a result, the EAI tool cannot move all of the information into the ERP, only summaries and selected data. This means that for the functional area that is treated by that module at least some of the detail users are interested in does not exist in the ERP. And if it's not in the ERP, it won't be picked up by the pre-built solution.

Diverse and specific report requirements

Its amazing how the layout (column order, summarization, which key performance indicators where) of a report can be very very important to report consumers. Its also amazing how much work can be required to modify the hundreds of reports and cubes that came with the pre-built system. Report writers everywhere are nodding their heads. Enough said.

Data Quality Issues

Even if you've used all the tables and fields in the ERP exactly as needed, and the reports look just fine to your users, if they have not already been given access to all this information you will probably find that there is lots of data quality work to be done. This cost is there for both the build and the buy options- just be aware of it, because it means the buy option is going to be more than just the sticker price.

One ray of hope- very specific industry solutions

Hold on, the vendors say, but we have industry specific data warehouses and data marts- these are built just for your industry so they know your business. We have one for banking data marts, another one for pharmaceutical data marts, you can get a sales data warehouse specifically for retail, and so on.

Without a doubt the more focused the pre-built data warehouse or data mart is the more likely it is that its going to save you time and money.

Conclusion? Often when something sounds too good to be true...

The bottom line is, although you can tell I'm not head over heals for prebuilt datawarehousing, I think the field has evolved. But then again, so have the prices.

The key is to be realistic regarding how much modification will be required now, and going forward, and how many of those hundreds of cubes and reports that are included are actually going to be applicable to your business.

It is possible that a package exists that will provide the best value. Just be very very careful when you are comparing the costs- because its never quite as sweet a deal as it first seems.

Smaller, very focused pre-built data marts, particularly targeted to applications or modules that are used fully by an enterprise are probably much lower risk and more likely to provide real value than a massive "out of the box, install it in weeks not years" data warehouse.

My personal experience with pre-built products was that in the end, when the dust settled, we had touched almost every ETL job, and the reports that were the most used and useful were ones we had built from scratch, largely using data loaded out of custom ERP tables.

If you have customization in your ERP, you have to think hard about this. The reason the custom tables were put in there was because they hold information that was important enough to justify the development. This probably means that the information in them is going to be important to the business intelligence you are building.

They have decided not to include a "Vista mode", citing lack of demand.

This is a very smart move on the part of Microsoft. Aside from the general bad press that Vista got (making users less likely to embrace a change), one of the key things that stops an IT department from recommending moving to a new operating system is the concern that certain desktop applications won't run on the new version. XP mode offers a way to ensure that legacy applications will ALWAYS run on this new version of the OS. Of course, in software "always" should perhaps never be capitalized, but running on a virtual machine its highly likely that your desktop apps that run on XP will still run on Windows 7.

Of course, the key here is not the technology side- its the fact that the XP license will be included in the Windows 7 license. You could have run a VM on windows 7- but no company will buy double the OS licenses it needs. Microsoft has just lit a rocket under Windows 7 in my opinion.

I imagine right now across the world IT departments are convening meetings to plan their Windows 7 rollout- and arguing about if they wait for service pack 1 or not.

Excel 2007 Adoption

Although good numbers are hard to come by, it seems like Excel 2007 has suffered a similar fate to Windows Vista, becoming the version for which people decide "we'll just skip this one."

Looking at a number of different sites that have surveys or discuss general estimates in terms of Excel 2007, and using a completely unscientific method of making up numbers that seem to be about the average, (see chart) we can see the early adopters, about 25% that use only Excel 2007. Then 25% percent use both (perhaps because some of their collegues are in the first 25%, so they have to be able to open those @#@*%#@ Excel 2007 files). And fully 50% are content with their previous version.

What is particularly interesting about these numbers (keeping in mind that my data sources and particularly my methods are suspect) is that they seem to be relatively stable for the last year. Regardless if the sources data was recent, six months or a year ago, it seems that the early adopters moved, and now everyone else is waiting. Waiting for the next version? Waiting for the Windows 7 upgrade that is coming?

The interesting question is "Will the next version of Excel entice them over?". We will find out in 2010. I'm going out on a limb here, but I don't think its a question of the new versions not having enough features. We're seeing a continued evolution in the business intelligence space, including desktop applications such as datamartist because existing tools just don't do what people need, but obviously spreadsheets are a more mature market. Of course Microsoft is also positioning Excel to be more of a desktop BI tool (in many ways it already is) but at the core its still a spreadsheet, and people know what they want from a spreadsheet.

Will Excel 2003 become the longest running application ever? And will every future version of Windows have the older versions supported with a virtual machine feature? Will Windows 8 have Windows 7 mode, that includes XP mode? Questionable.

Legacy software has been a reality in the enterprise world for decades, and this is officially sanctioned legacy at the operating system level. It used to be that the realities of change forced a certain number of the legacy applications to be re-written and made at least the desktop environment refresh relatively quickly.

It may be that that is about to change.

Lets review what we know;

• Lots of people use facebook. LOTS. More every day.
• The people do not pay for this.
• The people do not want to see advertising, and don't click on it.
• It costs money to give people a free service. LOTS. More every day.

And what have people been saying about what facebook is really worth?

• When Microsoft invested $240 M in facebook in 2007, it valued the company at $15 "B" for Billion dollars. (although it turns out Microsoft got some sweet preffered shares so maybe its wasn't quite that high.)
• facebook CEO Mark Zuckerberg recently rejected funding at a $4 B valuation
• Facebook shares that change hands privately are reported to do so in the $2- $3 B range
• Word is that Mark has another offer for financing at the bottom end of that range, $2 Billion.

Now, I know that venture capitalists have their own math and process.

But if I were to give facebook $4 B of my money, I'd expect a 15% return on it. So I'd want a cash flow of about $600 Million a year. If I saw that cash flow, I'd say to myself "Yep. Thats worth the $4 billion I paid."

No, No! The venture capitalists say. We're not giving them $4 Billion dollars- we're giving them $100 million dollars, and saying that the company is worth $4 billion as a way to determine what percentage we get, and we're factoring in a higher rate of return for all the risk we're taking. (Brave souls these venture capitalists, but they do need to be rewarded for such bravery.)

Ah. But how are you going to make your money back?

"With the IPO. Then the investors will pay the 4 billion, and we'll cash out."

Which investors? "Main St." - maybe your pension fund. Only problem is that after you buy the stock in the IPO, there is no IPO in the future. So then facebook actually has to make the half a billion dollars a year plus profit for you to get a return on your money. Hmmm. I wonder if the market is really going to pony up $4 billion. There has been some stuff about the economy, and over valuation of things in the news recently.

And when will the huge profits start rolling in? Unclear.

In my humble opinion, if you have 200 million people using your service, and you can't make a pretty solid profit, you need to reevaluate what you're doing. Don't tell me that you haven't reached critical mass yet.

Facebook is currently some new form of venture capital transfer system- it takes investors money, divides it by 200 million and gives lots of people a really cool website to go to and talk to their friends.

"But this is social media- its new!"

Riiiight. I recall hearing the same sort of thing during the last tech bubble.

I think facebook should be rightfully proud of the enthusiasm their users have for their service. But it's a company, and it's a business, and I STILL don't see the four billion dollars yet.

UPDATE 2011- Kudos to Facebook- rumors are that they are on track to do $2B in earnings- this means in fact they are worth more than the $4 Billion discussed in this post in 2009. But AMAZINGLY now people are throwing numbers like $100 BILLION as Facebooks value now. The multiple just grows to eliminate any reasonableness I suppose (you can just substitute the new numbers into the above). It will be interesting to see where we are at in 2012 when current speculation is that Facebook will go public. Perhaps I underestimate Facebooks ability to simply have every person on the planet signed up, and capture some insane percentage of all advertising and media budget...

My inbox seemed to be full of Microsoft offers to introduce me to their BI solution. I was talking to a friend, and mentioned this, and he said "Yeah- got an invite to watch a Microsoft Business Intelligence presentation today myself." My SQL Server magazine arrived in the mail and the front cover was "Unlock the secrets of SQL Server 2008 reporting services". I logged into linkedin, and found a post in the Business Intelligence group from an evangelist at Microsoft, linking to microsofts BI blog and a new video they've made- The history of Business Intelligence. Now I do tend to listen to channels that have business intelligence content, I'll admit, but it seems like a bit more than usual.

The video is certainly doing the rounds. I liked it- its well done in a light-hearted way, and outlines many of the challenges business intelligence has faced over time.

The best part by far in my mind is just around the 6:30 mark, where the proliferation of Excel is addressed. It can be summed up as follows:

And led to...

This where we are now, in effect.

Microsoft has the application we all have to run to when we can't get what we want out of our expensive Business Intelligence systems.

The big question is do they have the solution out of the Excel head bashing? Well, in theory part of it is the Gemini project- so stay tuned for 2010.

While you're waiting, take a look at Datamartist now and see how an Independent software vendor is working towards practical solutions today.

There was a link posted on meta-filter today that sparked some lively discussion amongst the crowd. The Excel auto formating "feature" loves to scramble common genetic nomenclature.

It turns out that in the genetics field, common codes get converted to incorrect values regularly. One example given was the code for tumor suppressor "DEC1" which gets coverted to the date December 1. Another was the code "2310009E13" (apparently a "RIKEN clone identifier") - which would be converted to a number, 2.31E+19. I'm not a geneticist but I can just see how this wouldn't be helpful.

I checked these examples on Excel 2007, and sure enough, the default will make changes right at import that scramble the mentioned codes- no error, no notification, no problem. Of course Excel is perfectly capable of handling this data properly- the user needs to specify the field as text, and the conversions won't be done.
The key point brought up in the article (and is always true about excel spreadsheets) is not just that in this case the data gets corrupted but that depending on how carefully a user checks the error may not be detected.
If undetected, what decisions, conclusions and actions will be taken based on the incorrect information?

Excel is super powerful, and super useful, but we have to always remind ourselves to balance the ease of use with how critical our data is, and what the impact of errors might be. In the end, as with all computer use, we have to test, validate and test again at a level consistent with whatever use we are putting the data to.

Interestingly, there seems also to be a significant population who use both versions at once. (Its possible-  THANK YOU MICROSOFT- to install both versions at the same time.)

Being in the software development business, and since my company is a registered Microsoft Partner, I have access to lots of Microsoft software, for test and internal use purposes.  I have lots of different combinations of XP, Vista, Office 2003, Office 2007 with various options installed on my virtual machines for testing and design work.

But for my personal use? The long and the short of it is that I had Excel 2007 installed on my primary machine for about 48 hours. 

It wasn't that I didn't like it- I just didn't have a pressing need right then, and so I didn't want to spend the time and effort of learning the new interface- yet.  I will soon, as I move into the next phases and let the Datamartist tools take advantage of its new features.  But at this point I'm doing data transformation and management work- Excel is better at the presentation side- and it seems to me that that's where most of the new features in Excel 2007 are focused on.  I'm sure its awesome-  I'm working on tools for getting data in a state to be used in it first.

And I wouldn't be surprised if many of you are in the same boat.  You do all sorts of analysis in Excel, you have come to know its interface inside and out, and well, although you have nothing against the new way of doing things... its just different, is all.

And lets face it- unless there is a specific new feature some specific need drives you to, Excel 2003 is pretty powerful for what it does. 

But we can be assured that we are not alone in this resistance to change- some of the most popular add-ins for Excel 2007 provide retro-menus to avoid searching the ribbon.

So don't feel that you are behind the times- we'll all get there when it makes sense, and when there is payback- not just because its there.