The majority of the development cost in the current business intelligence methodology is often in getting the data out of source systems (Extract), and transforming it to make it consistent across all the various dimensions needed (Transform) and then putting it in a model that is easy to query and analyse (Load). The creation of these ETL jobs is made dramatically harder if the data in the source systems is not consistent.

Change is the challenge

Companies are not static- they grow, diversify, change strategies, reorganize, rename and restructure. They acquire other companies or are acquired. The structure and content of the data their systems often tells you this story, and if the proper work is not done to keep the data consistent with itself and the new situation then this story will be painful and complex.

Remember ten years ago when we acquired company X, but decided not to change their customer codes to our standard, so all the codes had an "X" prefixed so that we wouldn't have duplicates? Well, those X's are still there, and all our queries have to deal with multiple code structures.

Remember how we used to have three independent databases, one for each region, then when we went to the new data center and put everything into a single database, we ended up with multiple schemas and all those crazy views rather than consolidating into a single instance?

When the data migration project made the decision to reduce the project cost by not addressing data consistency, they simply pushed this cost in the future, most likely turning a one time expense into an ongoing and expanding annual business intelligence cost.

You end up with crazy ETL jobs that parse the same field in different ways depending on the date of the transaction, or on other fields- "If the transaction is before 2002, then the first digit of the product code means X, otherwise it means Y, unless of course its from the western division, who do it differently so then you need to look at field A and use the CASE statement..."

Reduce Business Intelligence cost through data cleanup

If your data is cleaner you'll reduce business intelligence cost across your entire BI architecture.

• Reduce ETL and report development cost- both initial, and the cost of ongoing maintenance. Every change request will take more time if all the models are complex due to underlying data complexity.
• Reduce hardware costs- complex queries require more processing, and bigger servers to meet that nightly load window
• Reduce time spent reconciling numbers. Complex ETL means that chances are business intelligence reports don't match up easily with the operational reports from the source systems. People will spend time constantly double checking these discrepancies, and it will undermine confidence in all data.

Fix the problem at the source. Not in the Business Intelligence.

Business intelligence is far too often left to fix all the issues in the source systems- and then becomes the focus of dissatisfaction when costs and delays become unacceptable.

I've heard people argue "Thats what ETL is for right? Why are you complaining?"

Assuming that the ETL will fix the sins of the source system is an inefficient and costly strategy.

Everything is a balance, perfection does not exist, but when deciding what to fix and what to leave, don't let a lazy data migration project saddle you with years of business intelligence costs- when it's time to bulk load data into the system, make it as right as you can.

Users taking the initiative with data has been seen as the enemy of a successful business intelligence program.

This needs to change. In a world of ever increasing data volumes and complexity, faster business processes and more data savvy knowledge workers, a purely centralized solution is doomed to fail.

A consensus is starting form that the best architecture is one that blends centralized with more distributed and (gasp) free form, user guided methods. In fact, when we look at what actually exists in most enterprises and take into account the unofficial shadow systems, we're already there, but in two separate camps that aren't talking.

The amount of freedom to allow ranges from letting the users have at it, to opening up the possibility of departmental data marts, but the buzz out of TDWI clearly indicates a growing acknowledgement that a rigid top down architecture is not tenable.

What are Oracle, IBM, Microsoft SAP and SAS (who own more than 70% of the Business intelligence market share) advising as being the right approach?

They advocate big architectures, centralized meta data management, big databases, lots of command and control. They talk about "self serve"- but they mean to existing reports or report interfaces. To be fair, they need to sell the tools they have.

For a refreshing change from this, I very much enjoyed reading Mark Madsens keynote at TDWI "Stop paving the cow path".

We enjoy reading things that we agree with, and I nodded my way through his slide deck.

In his presentation, Madsen points out that centralization won't work, because it:

• Creates bottlenecks
• Causes scale problems
• Enforces a single model

Bottlenecks and Scale

In a centralized system, all requests go into the queue, and the backlog starts piling up.

The size of the department/team that is responsible for making it all work becomes the number one bottleneck.

Are there enough people able to prioritize and analyse the payback on analysis requests? Because in a centralized organisation, the gatekeepers are necessary, and how do they KNOW which requests are the good ones? How does anyone really know?

I'm not sure any company can afford to staff a centralized data warehouse team to be able to handle all the requests as they are generated. Prioritization therefore becomes a single point of failure. Get it wrong, and it can be all wrong. In a more distributed structure, decisions are made at multiple points, some good, some bad, but diversity will often bring more innovative and experimental behavior, resulting in new avenues of analysis that a overly static central team might avoid.

For an indication as to how well users think the central team is listening to them, take a look at how many excel spreadsheets there are around, and how many shadow systems grow like mushrooms throughout the standard enterprise. People think their analysis is important, and even if IT won't or can't they find a way to try to get it done.

In terms of scaling, I can hear the technical types starting to explain about how their servers, infrastructure and approach scales- diagrams and MPP theories pulled out with pride. "Centralizing lets it be scalable- what are you talking about?"

Maybe. But there are traps here too- centralized organisations always want to put everything in one database. Having everything in a single repository starts to become the goal- not the cost efficient analysis of the right data. Not centralizing is very scalable- stand alone machines can just be added for ever.

It may in fact be that data can remain distributed and diverse at certain levels of detail, and more federated approaches can be used, resulting in cheaper hardware and software, and more importantly avoiding a lot of really hard master data management work. Consolidation can sometimes happen at summary levels that make sense from a business point of view- not just blindly following the "one version" mantra.

Enforcing a single model

Isn't having a single data model good? We've been told that it is. In a way, this is the holy grail.

But is there a single, correct, slowly changing model that satisfies everyone in an organisation?

Why do I say slowly changing? Because if there is only one for the entire enterprise, it will change slowly, if at all.

Even if you happen to understand what the right model is, (and by model I mean data model, analysis model, process model, any model) and you manage to implement it while its still the right model, in a year its not going to be the right one. And a centralized, high cost, committed architecture won't and can't adapt. You'll still be paying the mortgage on the data warehouse.

Very large centralized models cannot be comprehensive and up to date, because to be comprehensive they have to be so complex as to be difficult to change, and as a result they quickly become out of date. It's sort of a Heisenberg uncertainty principle for common meta data repositories.

"Giving people their flying cars"

Madsen of course doesn't solve the entire problem in his keynote, but he points out some directions that make sense. And his graphic depicting a happy couple blasting off in their very locally controlled flying car sends the message- users can do their analysis without central oversight or interaction. (Although, one would imagine that some sort of air traffic control would be necessary, and the refueling stations for the cars would probably be run centrally- we're not advocating anarchy here.)

Having built data warehouses, established a data warehouse competency center, and provided business intelligence services for thousands of users, I can testify from first hand experience that centralizing alone is just not going to work. People who worked with me a decade ago will remember the significant amount of time spent creating meta data repositories. Are they still needed? Yes. But they simply can't do everything. Use them with care, and be wary of your ambition for them.

First, accept the fact that users are not mindless consumers. Learn from the fact that they use excel constantly, and they don't just read reports- they build things, adding data, fixing data, re-organizing data. They think. Give them tools that include them as part of the data processing.

Business intelligence cannot not be solely a process where formal requirements are gathered, followed by a publishing exercise of delivering the reports on time.

Are there some reports where this is the case? Sure. Monthly management reports and dashboards shouldn't change every month. The model can work for some amount of the delivered data analysis.

The entire architecture isn't getting ripped out- but if the new architecture is successful in bringing the pent up demand that is currently being satisfied by shadow systems into the light, then distributed, user centric, user driven business intelligence will become a significant percentage of the total.

But the old way of thinking has to change. Don't "Crack down on shadow systems".

Find a way to provide better service, be it self, assisted or centralized service that makes the shadow systems simply a less effective way to do it.

The existence of shadow systems, and the extent of them, is the clearest argument that centralized business intelligence alone is simply not up to the task.

Once you have people doing whatever they want in the self directed part of your architecture, DO watch what they are doing- not to control it, but to learn from it. Everyone constantly re-structuring the customer dimension? Obviously it's time for an update. By watching what users edit, what gaps they fill in, you can find the data quality issues, identify the fuel to put on the self directed fire.

Tools like Lyzasoft, our own Datamartist tool, and Microsoft's Power Pivot in Excel 2010 and others are all going to drive power to the users, and introduce a new balanced approach between centralized and local parts of business intelligence architectures. Visualization tools like Tableau will further give people the ability to create powerful, consumable analysis in a self serve mode.

Will there be challenges with data quality, risk management and wasted time doing pointless analysis? Most likely.

Will the information we gather and the payoff from the successful bottom up analysis efforts make it hugely valuable overall? I for one think so.

We need to learn to trust our colleagues with the data, while at the same time managing the reality of data quality and risk of errors that more free form techniques can create.

Companies that include both top down and bottom up capabilities in their architecture will stop wasting time fighting internally, and start to take advantage of all that data.

• $150 000 is the AVERAGE cost of business intelligence software for a DEPARTMENT
• ETL software (Extract transform and load) is also $150 000 on average.

And the rule of thumb for cost of effort and services is 5 times the software cost

I'm not making this up. Check the link.

In the end, Boris suggests that the cost of a single, fairly straight forward report might be $20,000. Of course as he rightly points out there are lots of variables, and it's a classic case of "it depends", but even so- clearly you want to be sure the reports add value when you are using a process that requires that kind of investment.

Boris mentions in passing that the cost of a single day of an external developer he uses for estimating is $800 USD. You can buy two licenses of Datamartist and take a friend out for dinner for that.

Don't get me wrong- for a number of applications you need the big enterprise stuff- but in my mind it makes sense to avoid it when you can. Enterprise business intelligence has its place, but there are alternatives. The rampant use of Excel spreadsheets is evidence of the fact there is huge demand for data out there. Try Datamartist and find another even more powerful way to get a the data for those cases where you need to do more than a spreadsheet, but it's not time to kick off a data warehouse project.

In the data analysis realm, spreadsheet errors are often used to try to dissuade companies from letting their people "work with the data directly". Software vendors of all sizes, but particularly the really big ones (those incumbants) spread FUD because if they can stop people from getting at the data themselves, it increases the chance of companies buying some more business intelligence suites.

The argument goes something like this:

Spreadsheets have been shown to be plagued with errors, many studies showing error rates above 90%. You need to reduce the risk that spreadsheets are creating in your organization by establishing formal, documented processes that are created an managed by professionals using sophisticated tools.

Then the usual nightmare scenarios are brought out, all involving rabid Auditors, Sarbane-Oxley, governance failures etc.

Now, don't get me wrong, spreadsheet errors are a very real and serious problem, and there are all sorts of data applications that should never be done in Excel or other ad-hoc, user driven tools. Ever. Formal documented processes are critically important, and there are lots of places where you better be using the right tools and professionals.

I have seen the culture of the spreadsheet completely undermine initiatives that would have driven better data quality, data analysis and business processes. The spreadsheet certainly has its dark side.

But the problem is that FUD paints with a broad brush. People take it as "Spreadsheets with data in them? Bad news. Don't do it. Individuals able to get at the data, and quickly transform it, analyze it? Who knows what they'll do- shut them down!"

Sadly, from a data quality point of view, sometimes the spreadsheets have the BEST data quality- because people have fixed the issues they can't fix in the transactional system due to constraints or IT department delays.

Encourage positive change with reasonable controls.

Intelligent, responsible people should be encouraged to use "informal" methods and tools to do data analysis.

These people will find things, learn things, and drive positive change (including change in those big formal professional systems).

They should do it with a reasonable understanding that doing things in an informal way, with spreadsheets or other tools does introduce errors, and should consider this when they recommend taking action based on the results.

Balance between two extremes

The totalitarian state: I don't think there is an IT department in the world that is capable of stopping all unofficial data analysis. In fact, I would suggest that the moment such an IT department comes into existence, it would kill the host company, a harsh sort of self-regulation. People interested in data and thinking for themselves would just pack up and leave. So who would be left making the decisions and based on what?

The twisted web of spreadsheets: Companies that allow an anything goes, visual basic code, macros and manual cut and paste direct to the annual report environment are not going to be long for the world either. They populate the horror story pages on the spreadsheet risk websites.

The zone of win.

You want to be somewhere between insane spreadsheet addiction and strict formal big tool paralysis.

I submit that companies that balance risk while still encouraging their smart people to "play" with the data and do analysis in new and interesting ways with new tools are going to win.

Again, don't let this process generate your profit and loss statement- understand where and what the informal discovery process is for- but do let it discover things. If it discovers something interesting you'll have the chance to check for the errors. Make sure its part of the process to do so.

By letting the FUD get you down, you'll never get that far and who knows what insights you might be giving up?

Of course, we believe you should go even further and give those intelligent, responsible people new tools that are less error prone than spreadsheets but still provide as much or even greater flexibility. That's why we're building Datamartist after all.

Openness, balance, and clear minded pragmatism will get you further than FUD every time.

Our data, however, is often less dynamic. All those sales records in the old sales system don't morph into the format required in the shiny new ERP. The data has to be dragged kicking and screaming into the new format.

Thus, the art and science of data migration is one of the most challenging, and most important, of the information technology black arts. In the next few posts, I'm going to take a light hearted, but hopefully useful look at data migration.

So imagine you are in a company that, for whatever important, unavoidable, and "it was super clear at the time" reason is going to change its sales system. The new system is just so much better- as the VP of marketing points out:

"The new sales system will drive synergies and encompass our core strategy for value creation and customer focus."

Oh good.

What are the possible strategies in regards to moving your data?

Abandon your data

Sure, just shut down the old server, send the hard drive to the recycler, turn on the new system empty, and wait by the phone for the next order.

"Hello, Acme lots of products, can I help you?"
"You'd like to place an order, ok, can I get your company name and address please."
"Yes, I know you've got a customer number, but we don't use those any more, they're from the old system. We have a new one now."
"You want product 23432- that's an old code, let me try to find the new one, just a sec..."

Probably not going to work out so well.

Enter all your data by hand over the weekend.

Depending on the amount of data, you might actually be tempted to do this. Sit a bunch of people in front of a bunch of computers, give each of them a stack of reports from the old system, and show them how to enter the data into the new system. Buy them pizza. Look really stressed as 6am Monday morning approaches.

The "advantages" of this approach:

• Because the new system thinks that you aquired every one of your existing customers over a two day period your companys growth rate looks fantastic.
• You get written up in computer science journals as a case study on the error rate for manually keyed data. The researchers are particularly excited about the clearly defined ramp affect built in by the progressive sleep deprivation of the data entry clerks.
• You get lots of practice calculating customer abandonment metrics as over the next six months data errors cause undelivered invoices, mis-routed orders and incorrect product configurations. You get excellent data on what makes customers most angry, and drives them to switch to your competitors most quickly.
• Even though the new sales system doesn't work out so well, the reduction in transactions makes it possible to save lots of money by reducing the number of servers and the amount of storage needed in the data center.

Or, you could launch a data migration project

Yep. Probably what you should do. But how?

Seriously, what are some things to consider in a data migration project?

Over the next few posts, I'm going to continue on the theme of a sales system migration, and talk about strategies that can help get you through your project, and the kinds of "Gotchas" that can spring up. Every data migration project is different, but hopefully I'll shed some light on the process, and share some of the experience I've had.

For me, the number one thing to think about in a data migration project is the impact on your customers.

Remember those customer people? The ones that give us money and expect something good in return? Turns out they're the cornerstone of your business.

Minimize the impact on your customers

Now, I'm not saying that you shouldn't consider other stakeholders too, and there is always political and financial constraints to any large project, but if it comes down to a choice between making life difficult for someone inside your company, vs a customer- well, someone has to take it for the team, and the customer isn't the right choice.

A badly executed migration project can cause enough issues for customers to push those that were wavering to the competition. A really badly executed one can push loyal customers away.

Depending on how cut throat your industry is, it could cost you significant marketshare, maybe even end up being the start of the end.

On the other hand, a well executed migration to a system that not only cuts your costs but provides real value to the customer could be a key part of your competitive advantage.

Next post: A data migration project is never just a data migration project- it's a data quality project too.

I believe in an approach that firmly places people in the center of a new way of looking at the data analysis process.

In the past, “big business intelligence” created an infrastructure heavy, highly centralised and technology focused approach to getting data from source systems into reports in the hands of the users. Under this regime, users were not to be trusted with raw data, but were given tightly controlled, managed and aggregated reports in order to protect the “single version of the truth”.

The theory and practice were tightly defined, and had been honed over decades of business intelligence and data warehouse orthodoxy. Giving raw data to end users would lead to chaos. Letting end users define new ways to look at the data would corrupt the master data, and lead to everyone looking at something different.

You can guess the sort of response this "don't give them the raw data" approach gets from capable, curious people that want to get down to some real analysis.

But to be fair you can see why these concerns are thought to be well founded. Almost every large enterprise is awash in a sea of excel files and a tangle of links and formulas. Excel is a wonderful tool, but it only offers the illusion of solving the data transformation problem. It is a much better reporting/dashboard tool than an ETL. (Although in the right hands it can do remarkable things.)

And this is the true state of affairs now. When the “official” system does not provide the answers that the business needs the people who need to make decisions get the data anyway, and they do it themselves. They do it in excel, they take night courses in Structured Query Language (SQL) they hire consultants (or even summer students) to build rogue data bases that they run on servers hidden under desks to get at the answers they need.

It is easy for the data warehouse theorists to highlight the clear issues with "spreadmarts" and "shadow systems".

But we need to be pragmatic. The reality of building a centralized structure that imposes strict formal rules and change management processes is that often while it does ensure that there is only one version of the truth, it is a version of the truth that no one can use because it has been so formalized, aggregated, compromised and delayed that by the time it is delivered the pressing business questions have changed and meaning has been expunged. The data warehouse becomes reporting rather than analysis.

Its clear that enterprises need this kind of reporting- I'm not advocating abandoning the existing approach- but augmenting it. Up till now, the solution has often been "more of the same".

The regime decided that the solution was to add more technology to the central systems, increase enforcement, and search out and repress all the dissident data manipulators. The data resistance was forced to go underground, to hide their spreadsheets, to outwardly appear to be following the official line.

It is very true that there are some risks in allowing people to analyze their own data, but there is also a reward. There are a small group of people who love data, who understand the business questions, who work to tease insight out of a steaming pile of raw data and can find things that are game changing. Massive, formal, designed by committee data warehouses can deliver a powerful and useful view of things, but they rarely offer flashes of insight. When they do, it is often during the design and discovery process- rarely by users using the system after it has gone live.

The Datamartist tool has been built based on the belief that both formal, centralized systems AND local, personal data transformation have a place in the architecture and that both should be official places.

People can be trusted with the data. In fact I think for an organisation to truly be successful at mastering its information, they have to be.

We have to realize that we can't allow our obsession with the quest for a single version of the truth to turn us into totalitarian regimes, certain that OUR truth is THE truth, and that messing around with the data is by its very nature subversive and dangerous.

Data to the people.

This is interesting, because often when we are searching for something via, say, Google, we are actually looking for an answer. We do it in two steps- "Country population list"- which gets us the document, and then we look up the countries we are interested in.

Unfortunately, the way Wolfram Alpha was launched and the way the media and observers in general tended to react has created a fair amount of hype, and misconception. Although Wolfram Alpha (lets call it WA) will have a natural language interface, people always get carried away with their expectations for such things. I'm certain that WA will be impressive, but I'm equally pretty sure that you won't be able to say "Roughly how many people like to have peanut butter on their toast in Ohio" and get a reasonable answer.

In a recent interview with Rudy Rucker, Stephen Wolfram said that rather than sell WA to the search engines, “We’d rather look for things like partnerships or licensing deals or APIs. I see a new field of knowledge-based computing. Imagine a spread sheet that can pull in knowledge about the entries.”

Now this is really interesting. What if it has a way to ask questions like "what is the GDP of [country]" just like that? What if it can tell you the population of any given Zip Code? What if it knows the rate of income change by county? What if it can tell you for any geo-location/date if it was a statutory holiday or not on that day? These are things that could be very very useful in doing data analysis- and are the kinds of things that can build interesting real world dimensional tables in a data warehouse or data mart.

There are of course, various sources to get this information now- but if one, massive, super flexible, broad "answer engine" existed- this might be a real boon to business intelligence practitioners.

Imagine generating a dimensional table by accessing the web service and enriching what your business users can analyze- and knowing that the values are as up to date and as accurate as the brain trust at Wolfram Research can make them.

Although there is some question as to if the API will really be a focus its clear that there is some interest in business applications. Its just not clear if Wolfram research shares this vision.

Although the masses that are expecting to have a conversation with HAL will be disappointed, there might be a new resource in the world for dimension building for data warehousing- I will be following this with interest.

Knowing this, it's not surprising that a huge amount of data collection, extraction, and transformation happens in Excel spreadsheets, or Access databases that are made without the involvement (and often under the direct scorn of) the IT department in large companies. In my previous life I was in the IT department, and I saw some amazing systems generated with hundreds of spreadsheets and databases. This mix of spreadsheets and databases, created without the involvement of the IT department by power users or external consultants (financed out of departmental budgets) are often referred to as Spreadmarts or Shadow Systems.

For an interesting survey on the subject, take a look at TDWI's report "Strategies for Managing Spreadmarts: Migrating to a Managed BI Environment". This report is now a year old, but I'm certain as valid as ever.

The title suggests that the solution is managed BI- I won't get into that right now, but you'll notice the study was sponsored by the likes of Microsoft, Cognos, Microstrategy and SAP- so of course the solution is Big Business Intelligence solutions.

But what's really interesting from the survey, is how the different groups within the respondent companies feel about spreadmarts and shadow data systems. The analysts love them, the executives are unsure, and IT hates with a passion. This makes for an interesting mix.

This is very much what I've seen in my experience. IT and the Business are at odds with each other, and senior management is either disinterested or forced to take sides.

Where do I stand? I'm in the "avoid them if you can" camp when we're talking about a tangle of spreadsheets and undocumented MS Access databases that can be error prone and time consuming. I understand why it's often unavoidable, but I've seen first hand how painful these systems are to maintain.

On the other hand, I don't subscribe to the school of thought that says "Excel needs to be eliminated- analysts should use the Business Intelligence systems only, otherwise there will be chaos." Let's not go overboard. Excel and spreadsheets are useful tools, and have their place. Additionally, I really feel for business users who simply can't get what they want from the IT departments. I used to be the IT department, and it was frustrating to not have the resources available to build what people needed.

As one of the authors of the above report, Rick Sherman, said in a recent podcast:

"reality is no matter how many IT folks that you have in your company you're not likely to have enough resources or time to meet every business users reporting or analytical requirements..."

He presents what is a refreshingly balanced approach to Excel. In his predictions for trends in 2009, number 5 is "Excel becomes an accepted tool in a BI portfolio". He points out that this may not be mainstream in 2009, but I hope he's right about the trend. A pragmatic, inclusive strategy with more power to the people while avoiding the chaotic side of spreadmarts is where the solution is.

Check out the first four parts of this series (1,2 , 3 and 4) where we created an example data mart, with some fictitious purchasing data.

The final step is to join the dimensions we have created to the fact table. To do this, we connect up the two dimensions (Vendor and Item) to the Join block and connect an export block to the output. What has in effect been created is a complete Extract, Transform Load (ETL) and the final star schema join.

(If thats a bit hard to read- click on the image to see the full size screen shot.)

With the generated data set I used for this example, summarizing the data to yearly totals but keeping all the detail on Vendor and Item causes the roughly 4 million row raw data file to be reduced to around 800 thousand rows. (This summarizing was done on another canvas- although it could have been done on this canvas just as easily).

This data mart, with 800 k rows and two dimensions of about three thousand members each took my laptop about a minute and 45 seconds to solve, and save to a 360 Mb text file out.

Of course, by summarizing or filtering (just add blocks) analysis subsets could easily be exported directly to Excel, managing the data volumes involved, and letting you create the graphs, dashboards and reports that you need.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5

Now we need to create a Vendor dimension table and join it to this fact table to determine who our big vendors are.

In Datamartist it is a simple task to create this vendor dimension. As always we use blocks and connect them together. We define a dimension by using a reference definition block. All we have to do to configure the reference block is to specify which columns uniquely define the dimension (or almost uniquely, Datamartist will resolve duplicate keys using a majority/first rule set for you if you have some data glitches).

We start with an import block that brings in the Vendor master text file, then we define the reference by specifying "Vendor_ID" as the key. These first two blocks look like this:

Then we join it to the fact table we created in part two of this series with a join block. This means that now instead of just the vendor ID number that was in the fact table, we have the name, and address for the vendor in our mini star schema.

And finally we put a summarize block after that to total up all the monthly values for each vendor, and we export to excel. This is what the canvas looks like:

After we do this, we grab the excel file Datamartist just created for us, do a quick sort, and come up with a list of Acme's top ten suppliers. Feeling pretty good about ourselves, we do a review with the head of purchasing.

"Where's Mega brothers?" she says with a frown "I think your data is screwy- no way that Mega brothers didn't make the top ten- we spend a fortune on railways, and a lot of our freight goes with the Mega Brothers Rail company. Of course it is probably entered under different vendors, each location works with the office local to them... But we've got to view them as a single vendor in the data mart- you can do that right?"

Fixing Duplicate Rows

Having to deal with duplicate data is a very common issue in any type of data analysis. So, back to the canvas. By simply adding a de-duplicate block to our Vendor dimension table (after the Reference block, and before the join) we can find and resolve the Mega Brothers duplicates.
We just use the filter to find the records- (Easy to do, looking for "Mega" "rail" "brothers" etc. and we map them to a single instance.) This is the filter control that lets us find and tag the duplicates:

As we tag them, they show up in the mapper, which lets us see which duplicate records we have eliminated for the dimension. We run the canvas again, and this time, sure enough, Mega Brothers Rail is in our top ten. But even though the head of purchasing knew it was a lot, this is actually the first time she's seen the number. "Wow. I've got to give them a call- can you give me that in an Excel spreadsheet?"

Stay tuned, more to come as we go further into Datamartist's ability to segment, filter and organize large data sets.

If you want to see the interface in action watch our first Tutorial Video. Or just get right to it with your own data- download the free trial now- there is no registration required, and it installs in minutes.

This is part of a 5 part series- here are the links to the various parts: 1,2 , 3 , 4 and 5