Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.

A bit of googleing led me to the blog of Will Scully-Power, who lays claim to coining the term "datarati". After a bit more surfing, its clear that there is buzz around the concept of data digging in the online advertising world, and particularly the new datarati that are driving it, and bringing the data analysis skills that are needed to the table.

In a recent article the New York Times reports;

Major advertising holding companies like WPP, the Publicis Groupe, Havas, MDC Partners and the Interpublic Group are starting data practices, hoping to latch onto what is expected to be the fastest-growing category of online advertising in the next five years.

What these new datarati know is that closed loops are the way to go- quickly gather data, analyze it, and make decisions and take actions that will affect the process. Repeat. Optimise. Move faster because you know you can adjust on the fly. Find entirely unexpected, unplanned success because you can move fast enough and try enough things that serendipity can work in your favor. Don't brainstorm 500 ideas and then cut them down to just five that actually get implemented- do hundreds of them, SEE which ones take off, and iterate from there.

Consider the difference between online advertising and how television advertising used to work.

Television

1. Spend millions on focus groups, testing and validation
2. Book and commit to your ad space months in advance
3. Spend millions on producing a fixed, can't change it on the fly set of ads.
4. Run the ads, and pay the television network the money with no ability to change regardless of effect- (except canceling them, thus losing your investment)
5. Find out months later how effective they were (if you are lucky), but with only sketchy links between the ads, when they ran, and the demographics involved and any actual consumer purchases.

Online Advertising with the datarati at the helm

1. Quickly come up with multiple- perhaps hundreds (using combinations along dimensions) of ideas and ads
2. Put them online, tracking where, when, who, what
3. Based on the actual real time click through, and online purchase results modify the ads, where they run, change your bids in real time auctions and optimize the ad campaign return literally minute by minute.
4. Data mine the results to find patterns and identify new sub-groups of customers, test hypothesis with new targeted ads.
5. Drop what's not working, add new things, repeat, repeat repeat.

The datarati will have hundreds of iterations with real feedback from thousands of customers done before the old advertising model would have invited in the first fifty people for coffee, donuts and a focus group.

The famous "Half my advertising budget is wasted, I just don't know which half." has changed to a world where the datarati can know exactly what has to be cut, and how to optimize the rest, with granularity down into individual consumer/ad interactions.

It's exactly this closed loop, large data volume, modify on the fly as needed data analysis that gets me excited, and is the whole concept behind the Datamartist tool. Datarati are people that are passionate about data- want to see it, touch it, and try things with it themselves, not content to wait for others to build something that may or may not still be needed.

The datarati point of view is "if you aren't using data to make decisions, what ARE you using?"

The datarati are going to be using personal tools, they will be messing with their data themselves. Lets hope some of them find their way here, and give Datamartist a spin.

The first one talks to the question of what the perfect BI tool is- answer? One that people actually use. The second uses a story to illustrate that many uses of information to make better decisions are business intelligence- even if one of the Mega tool suites isn't used, and there is no massive data warehouse in sight.

Both of these blog posts remind me of my days in the business intelligence supply side of a large organization, where I was tasked with creating, maintaining and evolving the "official" business intelligence systems. Anytime I gave a presentation to end users (often a demonstration of a new tools capabilities and power) the same question would always come up- Can we export data into our spreadsheets?

Everytime I showed an export to Excel functionality, the release of tension in the room was noticable. Everyone relaxed. They could get their numbers out, they could take charge themselves, and could get the information they needed. We worked hard to try to provide the functionality in the core systems, but the reality is that the needs of the users changed, often more quickly than we could make our massive databases, ETLs and SQL structures evolve. We were beholden to our change management procedures, we had to analyze the impact of changes on the whole.

At times, it was suggested that we "crack down" on the proliferation of spreadsheets and home made MS Access databases that sprouted when a need couldn't be met by the main system. But the reality is, the best solution is to work with the analysts- provide them new tools, understand their needs and don't rule out solutions just because they do not follow the "standard" business intelligence methodology of servers, big integrated suites of tools and rigid IT department involvement.

As with all things in life, balance between the "proper" way to do things, and a pragmatic approach that values individual initiative and results is important. After all, it's the only way what is "proper" can ever evolve and improve.

Check out the first four parts of this series (1,2 , 3 and 4) where we created an example data mart, with some fictitious purchasing data.

The final step is to join the dimensions we have created to the fact table. To do this, we connect up the two dimensions (Vendor and Item) to the Join block and connect an export block to the output. What has in effect been created is a complete Extract, Transform Load (ETL) and the final star schema join.

(If thats a bit hard to read- click on the image to see the full size screen shot.)

With the generated data set I used for this example, summarizing the data to yearly totals but keeping all the detail on Vendor and Item causes the roughly 4 million row raw data file to be reduced to around 800 thousand rows. (This summarizing was done on another canvas- although it could have been done on this canvas just as easily).

This data mart, with 800 k rows and two dimensions of about three thousand members each took my laptop about a minute and 45 seconds to solve, and save to a 360 Mb text file out.

Of course, by summarizing or filtering (just add blocks) analysis subsets could easily be exported directly to Excel, managing the data volumes involved, and letting you create the graphs, dashboards and reports that you need.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Having a way to create and manage tree structures (Hierarchies) with your dimension and fact tables is a key part of making a dimensional model in any data warehouse or data mart. Hierarchical structures lend themselves to managing a very large number of categories and we use them to create drill down paths.

Check out the first three parts of this series (1,2 and 3) to see what we've done so far.

In this installment, we will make a another dimension, the Item dimension. This will illustrate how the Datamartist tool allows you to quickly and easily generate hierarchies, and even edit and manage them in a graphical user interface.

The head of purchasing for Acme has asked us to analyze the company's spend on computer equipment- "I have a feeling some offices are spending more than others- but I don't have the numbers to back it up. But I don't want you to use the categories in the source system- I just want it broken down by Desktops, Laptops, Printers, PDAs and other. Can you do that with the data mart?"

In their source system, Acme is using the United Nations Central Product Classification, (UNCPC) and so we know that all the computer spending we're interested is in division "C45 Office accounting and computing machinery". The way the codes are structured is they have a code like "C45222", so we want to take all codes with the left three characters being "C45". We can do this easily with a filter block. After the filter block we connect a define reference block (to make a dimension), just as we did before-and finally, since we're looking at hierarchies, we'll add a recategorise block too- that last block in the chain is what we use to change the drill down structure;

Tree structures simplify alternate categorisation

The advantage of using a tree structure is we only have to rearrange the level of the hierarchy that encompasses the level of detail we need: we don't have to map each individual product, just the higher levels. So it's much less work to start, and when new products are added in the source system, they will automatically map up into the new categorization. Recategorising in excel often means search and replace at the bottom level which can cause errors, and has to be redone manually every time the data is updated.

When we open the recategorise block, we simply pick the levels we want to see, and then are presented with a tree view that shows us the hierarchy, automatically generated from the underlying data.

Now, directly within the hierarchy we can edit categories, add new categories, and drag and drop categories around to build the new drill down that we want. The interface is a lot like the windows file explorer, just like renaming and moving folders, except that you are building dimensional data. Of course, the underlying input data is not changed, so there is no need to modify the source system in any way, but the datamartist tool records all the mapping and is able to reproduce it when new data arrives.

You only have to edit the Hiearchy once, and from that point on your analysis can use both the existing, and the edited tree structure. It's possible to create as many different hiearchies as required- it's a fast way to do "what if" analysis, trying out different drill down paths and categorisations.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Now we need to create a Vendor dimension table and join it to this fact table to determine who our big vendors are.

In Datamartist it is a simple task to create this vendor dimension. As always we use blocks and connect them together. We define a dimension by using a reference definition block. All we have to do to configure the reference block is to specify which columns uniquely define the dimension (or almost uniquely, Datamartist will resolve duplicate keys using a majority/first rule set for you if you have some data glitches).

We start with an import block that brings in the Vendor master text file, then we define the reference by specifying "Vendor_ID" as the key. These first two blocks look like this:

Then we join it to the fact table we created in part two of this series with a join block. This means that now instead of just the vendor ID number that was in the fact table, we have the name, and address for the vendor in our mini star schema.

And finally we put a summarize block after that to total up all the monthly values for each vendor, and we export to excel. This is what the canvas looks like:

After we do this, we grab the excel file Datamartist just created for us, do a quick sort, and come up with a list of Acme's top ten suppliers. Feeling pretty good about ourselves, we do a review with the head of purchasing.

"Where's Mega brothers?" she says with a frown "I think your data is screwy- no way that Mega brothers didn't make the top ten- we spend a fortune on railways, and a lot of our freight goes with the Mega Brothers Rail company. Of course it is probably entered under different vendors, each location works with the office local to them... But we've got to view them as a single vendor in the data mart- you can do that right?"

Fixing Duplicate Rows

Having to deal with duplicate data is a very common issue in any type of data analysis. So, back to the canvas. By simply adding a de-duplicate block to our Vendor dimension table (after the Reference block, and before the join) we can find and resolve the Mega Brothers duplicates.
We just use the filter to find the records- (Easy to do, looking for "Mega" "rail" "brothers" etc. and we map them to a single instance.) This is the filter control that lets us find and tag the duplicates:

As we tag them, they show up in the mapper, which lets us see which duplicate records we have eliminated for the dimension. We run the canvas again, and this time, sure enough, Mega Brothers Rail is in our top ten. But even though the head of purchasing knew it was a lot, this is actually the first time she's seen the number. "Wow. I've got to give them a call- can you give me that in an Excel spreadsheet?"

Stay tuned, more to come as we go further into Datamartist's ability to segment, filter and organize large data sets.

If you want to see the interface in action watch our first Tutorial Video. Or just get right to it with your own data- download the free trial now- there is no registration required, and it installs in minutes.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

• Item_Master.txt - this holds all the items that Acme buys
• Vendor_Master.txt - this holds a list of all the vendors, with information such as their address
• PO_Detail.txt - this is the huge data set, all the purchase order data for the last four years

The Item and Vendor files aren't very big, but the PO_Detail is over 340 Mb, and it holds almost four million purchase order lines. Don't try to import it into Excel. Of course you need Excel 2007 to even try to import 4 million rows. In Excel 2003 it would take over sixty sheets and probably some VBA code to try it. I tried the import in Excel 2007- it takes 20 seconds just to tell me I'll have to go back to the text file import multiple times to do multiple imports onto separate sheets. It took almost two minutes to do the first million rows. Even once we have the data spread across four sheets it's not clear how to summarize millions of rows in excel easily.

Instead, let's use the Datamartist tool to manage this data set and generate one thats more useful.

The first analysis we will do will be on the Vendor dimension, to determine who Acme's big vendors are, and if we can negotiate some price reductions where we have leverage.

In Datamartist, very large files are not an issue because the tool can load in only preview data- this means that it's possible to look at a sampling of a few hundred thousand rows, and design the transformation before running it on the whole data set.

The PO Detail file has the columns shown- let's answer the question - "Who are our biggest suppliers"?
So which columns do we need? We probably want to have some sense of trends over time so we'll keep the order date, but summarize to Month, we'll keep the Vendor ID of course, and then we need to use the Quantity and Price fields to calculate the total amount spent. Then we want to write this summarized data into Excel to check it out.

To do this in Datamartist all it takes is four simple blocks; A Text import block to load in the PO_Detail.txt file, a calculate block to multiply QTY by PRICE, a Summarize block to do all the summarizing, and an Excel export block to generate the excel file;

Each block passes its result to the next block via the connectors, and the last block saves it to an excel file we've specified.

Defining the calculation uses standard spreadsheet functions- here's what the config area looks like;

And defining the summary is as simple as it looks- pick the columns you want, and select what kind of summary you want done.

We run it on a preview set of 100 thousand rows (takes about twelve seconds to run), and check the output.

It looks good, so we run on the whole 4 million rows;

About seven minutes later we have our result- an excel sheet with a manageable 130 thousand rows, total spend, by vendor, by month for four years;

Next up we need to create our vendor dimension, and join it to this mini fact table we have created. Stay tuned.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

In the next series of posts I am going walk through the design and implementation of a purchasing data mart, including its fact tables and dimensions to allow us to analyze some typical purchasing data. I’ll build this data mart model using the Datamartist tool.

This will create a “snapshot” analysis of purchasing data with a desktop data analysis tool that can be built quickly yet will access millions of rows of data, and deal with data quality issues such as duplicate rows.

For the purchasing data mart model that we’ll be defining, I’ll use a fictitious company that manufactures and sells a broad range of things- the "Acme does everything company".

Acme is a long standing enterprise, with a number of offices and factories in the US. But they’ve never done an in-depth analysis of their costs because they didn’t have to until now- profits were good, and the business was growing well. But then the economy took a turn for the worst, and Acme’s customers are cutting back on pretty much everything. Acme’s CFO has announced that if costs aren't reduced quickly, Acme is going to simply run out of cash. He wants you to head up the analysis on the company’s purchases- where can Acme save?

I look forward to showcasing the functionality in the Datamartist tool that makes it possible to do this without programming, and without requiring database software, developers or servers. This kind of snapshot, immediate data transformation is what we think will make Datamartist such a cost effective and efficient addition to any serious analyst's toolkit.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

With data marts (just like leasing a car) you get behind the wheel quickly, and it gets you where you want to go in style.  And the monthly payment is something you can afford now.  However, long term, well, in three years you don't own it, and have paid a bundle.

But let's be realistic.  Just as having all the cash on hand to buy the car outright just might not be in the cards,  a true data warehouse might require a very significant outlay before anything comes out the other end, making it unaffordable.  A quick, focused departmental data mart could be delivering value in a matter of weeks with relatively little investment.  (Your actual mileage may vary- depending on where you're at, its always dangerous to believe someone when they say "a matter of weeks" when software and people are involved.)

Will that departmental data mart, or even a number of data marts lead you to a single version of the truth?  Will it give you deep competitive advantage through a culture of data analytics and cross enterprise master data management? In my honest opinion, No.

But is it something you can afford in today's economy, and will you learn things about your data, your company's information culture, and your business that will be useful if in the future you embark on a true data warehouse initiative.  Yes.  Yes it is, and yes you will.

And I'll take it one (blatantly promotional) step further.  Is a personal data mart on your desk top as good as a full fledged departmental data mart with an army of highly paid developers maintaining it?  Probably not.

Is the personal data mart on your desk basicly free in comparision to the servers, software and hired help the data mart requires?- Yes. And does it, just like the data mart does for the data warehouse, prepare the ground for the next evolution when the economy turns around? Yes. Yes it does.

In difficult times companies that are pragmatic, and do what is possible, preparing for the day when more will be, survive to see that day.

It seems obvious that doing nothing because you can't afford to do the best thing is a bad strategy- but we need to ask ourselves, how often do we make that exact choice through inaction?

Although its true that in today’s economy, a lot of the multi-million dollar business intelligence projects are going to be cancelled or delayed, cheap data analysis is possible.

You just have to create cost effective data marts.

How can you reduce the cost of data analysis, cut your reporting costs, and avoid expensive business intelligence mega-projects?

Focus your efforts on Actionable Analysis

Before specifying a report or dashboard or data mart ask yourself “what action will I take based on what I see in this analysis, and how will that action move the business forward?”   If it’s not clear that some action can be taken based on the results, chances are there are better uses for the cash you’re planning on spending to get the report.  Target cost analysis first.  Reducing costs shows an immediate, verifiable return on investment.

Reduce the data scope as much as possible

Only analyze what you have to, and keep an eye on bang for your buck.  Don’t spend 80% of your budget cleaning up 5% of the data unless you really believe that data has something to teach you.  Eliminate dimensions from your data marts- really ask yourself if a dimension in the star schema is “nice to have” or critical.  Just because the data is there does not mean its worth the effort.

Reduce the report scope as much as possible

Don’t build reports no-one, or only few people use.   Consider providing an export function, so people can create their own reports in spreadsheets, rather than have hundreds of reports included in the scope of the project.

Use Desktop, Snapshot and One-time Analysis rather than full blown server based scheduled systems

There is a huge cost difference between a one-time analysis of a static data set, and a dynamic system that is able to load data daily or monthly from a transactional system automatically.  In many applications, it's the first analysis that gives you the insight.  Spending a lot of money to have the data mart refresh every day might not make sense if the first run gives you the majority of the information you need.

Obviously, its impossible to know where the insights really are- if we knew that we wouldn't have to do so much analysis.  But when resources are tight, you have to pick and choose.

Use the Datamartist Beta to create personal data marts on your desk

Sorry, I just couldn’t resist-  Datamartist is going to create a whole new price point for data marts.  If your pet business intelligence project has been delayed or canceled thanks to the economic meltdown, give Datamartist a try.  You’ll find its remarkable what can be done without programming, servers, or expensive consultants.  Once you've created a data set, you can export the information to your favorite data analysis or visualization tool and find those dollars to save.

Why do so many people turn to Excel for their analysis needs? Because with excel they are in control.  With Excel it IS personal. Massive, expensive data warehouses are one size fits none. Excel is the ultimate in customisation. 

“Despite years of investing in BI, many IT organisations have difficulty connecting BI with the business, and to get business users fully involved and out of the ‘Excel culture’”
- Bill Hostmann, Vice President and distinguished analyst at Gartner

Excel isn't always enough

Many users frustrated by the limitations in Excel in terms of data volume and functionality want more.  Some of them take up programming macros- others spend money they'd rather not on throw away database development.  But demand encourages supply, and new tools are beginning to emerge.

As Ted Cuzzillo put it so well in his trends for 2009 blog post recently:

“on business desktops, new, highly individualised tools will sprout.”

The Datamartist vision is that of a user driven tool that provides the flexibility of a spreadsheet with the key functionality of a database without the database. 

Personal data marts can for the first time be built directly on the desktop, without programming, yet handling millions of rows of data, and implementing truly useful data analysis.