CYBAEA Journal

When Big Data Matters

2012-03-21T16:14:00Z

Get this article as a PDF: When Big Data matters.

Big can be a qualitative as well as a quantitative difference. The gas in the ill-fated Hindenburg airship, the gas that formed our Sun, and the gas that formed the Milky Way galaxy were just lumps of hydrogen atoms (with varying impurities). The difference was in the number of atoms. But that difference in numbers made the three structures into different things. You simply cannot look at them in the same way. If you try to model the galaxy in the way you model a balloon you will fail.

Just a bunch of hydrogen atoms: when “big” makes a qualitative difference.

Back to planet Earth, the early years of this millennia saw an explosion of popular books and articles about the science of networks, which was, and continues to be, a subject of intense research. Networks are interesting, but the main point we will take here is that the network is fundamentally, qualitatively, a different thing from its constituent parts, and it can not be understood from just understanding the parts.

The canonical example from this time was inspired by the widespread power outage in north-eastern America on August 14, 2003, which affected an estimated 55 million people, then the second-most widespread failure of an electrical distribution network in history.

The components of electrical networks are well understood. Copper wires, transformers, generators, and the like have been studied for decades, if not centuries. Mr Maxwell had published the necessary mathematical equations by 1862. We know how to solve them and our calculations agrees with what we observe to a ridiculous 10 decimal digits, the limit, it should be noted, being our ability to observe accurately, not our ability to calculate precisely.

But put all of these components together in a network, and we are unable to really understand or predict extreme events in the system, such as the cascade that led to the 2003 outage. (I say this with with the greatest respect to the people building and maintaining power grids who are some of the most awesome engineers I have ever met and worked with. The failure to understand is because we are looking at the wrong things and because the right things are inherently difficult.)

The human brain is a network of relatively well-understood nerve cells, and yet we do not understand consciousness, perhaps the most obvious feature of our minds. The weather system can be considered a large scale application of the well-understood laws of thermodynamics, but long term weather forecasts are still beyond us.

The list goes on. The network is fundamentally different from its parts.

But you can just sample….

One objection to Big Data we often hear goes something like this: “If you select a sample of the data, and the sample is good and truly random, then you can do all your analysis on the smaller set thereby avoiding all the problems of Big Data.”

It is true that for many problems you can indeed sample, and it is a well understood statistical technique. If all you are doing with your data is a generalized linear model on a few categorical variables, then go ahead and sample to your heart’s content.

But how do you sample a network? We did some work with a mobile telephone company where we were interested in understanding if, and under what circumstances, when you leave the network for a competitor your friends leave with you. The data was somewhat over 10 million current and recent subscribers with information about their usage, spend, and “calling circles”: who did they call and how much, which was our definition of “friend” for this purpose.

We found that there were tipping points when a predicable and quantifiable part of a sub-network of customers had left, a large proportion of the remaining network would also leave. Intuitively you can understand why: if your friends or colleagues are all with this other company and love it and get cheap or free calls to each other, but not to you, then there is both a social and economic pressure for you to switch.

Consider if we had taken a random sample of customers, say 1 million, for our analysis. Now the network of friends is typically no longer connected and therefore not a network we can identify. To see why, consider a naive (and unrealistic for this application) network where each person is connected only to his two neighbours, as in the figure below. In this scenario, A and B are connected through two chains of nine friends.

After sampling, A and B are no longer connected

But if I take out 9/10 of the sample, then A and B appear very far removed. And if C is the trigger for the whole circle to leave, then I have lost my ability to predict.

To be fair, there are other ways of sampling a network. If we had looked not at customers, but at their calls we could have sampled those and rebuilt a reasonable representation of the network. But that means looking not at 10 million customers, but at perhaps 10 billion call detail records: hardly the way to avoid Big Data.

When Big Data matters

First, Big Data always matters when you are looking at the whole rather than the parts, and when that whole is a different ‘thing’ from the parts. A nation is more than a group of people, a city more than a bunch of houses. When you have data on people and houses but want to know about nations and cities then you are into Big Data territory.

Second, Big Data may matter when you are looking at the few rather than the many. In applications like fraud detection the signal may be so small that you miss it, or miss too much of it, if you sample. Whenever you are looking for ‘all’ or ‘every’ or for ‘the largest’ or some other superlative you are into at least large data territory. (If you are looking for the ‘mean’, the ‘typical’ or anything like that, you are looking at aggregates of the many, and you do not need Big Data.) The canonical ‘hello world’ example for Hadoop, a popular Big Data tool, is counting the frequency of every word in a text corpus; ‘every’ being the key word here.

However, for many practical applications in this second territory you can get away with some sampling or else the raw data just is not that big to start with. In either case, you are looking at what we might call ‘large data’, by which we mean data that is manageable on, say, a large computer or an in-house cluster using what is essentially classical analysis techniques and tools.

In summary, our rough guide to when Big Data matters look somewhat like the illustration above. When you are looking at the whole (and the whole is a different thing from the parts) you are in big data territory; when you are looking for the few among the many you typically do not need big data techniques though your data may be large; and if you are looking for small signals in the whole, you may be into truly huge data.

Business examples

Big Data may be relevant to you if:

You are looking at groups of people “spontaneously” acting together. Examples include predicting the next runaway best-seller that everybody suddenly wants to read or predicting which of a customer’s friends will leave when she does, as in our mobile telecommunications churn example above. This falls into the bottom right corner of our ‘When Big Data matters’ illustration. Pretty much anything “social” involving humans fall into this category, which is why Facebook uses Big Data in a (apologies) big way.
You are looking at the network and its structures or behaviours. We looked at discovering the top influencers for a company: those people who’s opinion about you, or your products or services, make a big difference on the general perception of your brand. We trawled through blogs, comments, purchases, interactions, and much more data to determine these key people and how their influence rippled through the network. Anything of this nature – anything involving the top end of a power law – is firmly top right corner of our ‘When Big Data matters’ model. Google is as much interested in how the web pages are linked together as in what they say, which is one reason Google is a Big Data pioneer.
- Another example is from our work on quantifying brand perception: you know who buys what and when, and you perhaps know what is being said about you in media and on social sites, but pulling all of that together to a picture of how your brand is perceived: that is a complex Big Data problem (but unlike the previous example not a Huge Data problem because we were just looking at the perception, not directly trying to identify the influencers).
Your data is at the level of the parts, but the structure that you want to investigate is more than the parts and you are not sure in advance how he parts interact. These are the houses/cities and people/nations type problems. Any random sample of the UK population would almost certainly miss the members of parliament (0.001% of the population) unless you knew a priori that they were important for your analysis and therefore skewed your sample.
- These problems are hard, because you may not recognise the extent of your ignorance of the behaviours in your data until after you have done the analysis.
- Examples include the power distribution network problem we mentioned before and much analysis in the social sciences.
We already mentioned fraud detection as an example of the top left corner in our ‘When Big Data matters’ model. They are large data problems because the signal is small and you can’t afford to sample until you have isolated that signal. However, they are often not Big Data problems because as soon as you have identified the small signal, you can sample the rest of the data for the comparison.
Where fraud detection becomes more interesting and the problem moves from the left towards the right of the model is when you are looking to identify collusion. This is also applicable for many compliance applications like insider trading or policing Chinese Walls within an organization. With one insurance company we looked at identifying staged accidents, but more successful examples come from the casino (Gambling) industry where the casinos successfully cooperate with each other to prevent fraud while still protecting confidential information about their “whales” and other customers.
Sometimes branded as ‘Big Data’ are problems where speed of response is critical. This is effectively a third dimension to our model, and where real-time responses are needed it can have a massive impact on the approach. We have much experience in augmenting call centre processes with real-time decision support tools but we will leave this subject for another note.

Dealing with Big Data

The reader will have noticed that we have so far avoided defining ‘Big Data’. For our purposes we use the following terminology:

“Big Data Sets” usually refer to data that is too large to handle using traditional relational databases and is inefficient to analyse using non-distributed applications.
“Big Data” is the industry of managing, transforming, and analysing these big data sets, and the ecosystem of tools and techniques that this uses.
Often not included in Big Data is the exploitation part which is our obsession: how to deploy the insights from data (big or otherwise) back into the business, its processes and its people, and execute on those insights.

There is not doubt that there is much hype about Big Data at the moment. But that just means we need focus. Big Data is real and addresses real business problems, though it is not a universal solution or requirement to every problem. This briefing should have given you a better understanding of the types of problems where Big Data will deliver business value.

Big Data is not scary any more. For sure you have to approach it different from “small data”, but we now have the tools, the infrastructure, and, perhaps most importantly, the techniques and the design and analysis patterns to handle very big data. And we understand how to exploit the insights from Big Data analysis within our business processes to deliver measurable value and sustained competitive advantage.

Until a few years ago, we had the excuse that Big Data was too hard and too expensive for all but academia and a few businesses. That is simply no longer the case: large and Big Data is for everyone. And those pioneers, companies like Walmart, Tesco, and Google who are the most successful in their industries, have shown us how data can be a sustained competitive advantage.

What is your Big Data opportunity? What are you waiting for?

Get this article as a PDF: When Big Data matters.

Jump to comments.

Commercial Analytics: The Capabilities

2011-10-05T21:43:00Z

Commercial Analytics is the kind that makes money. From data to dollars, insights to income, this is all about how to run the business better. To do it and to do it well you need certain capabilities in place. This article builds a map of those business capabilities to help you assess, understand, and plan your business.

Usually we talk about this and we are happy to talk to you about it (just contact us) but we recently had occasion to make a slide pack that covered some of the materials as a stand-alone presentation. This article is based on that pack which is also available as a PDF:

Download CYBAEA Commercial Analytics Capabilities presentation (PDF).

Commercial Analytics

Commercial Analytics is the kind that makes money. From data to dollars, insights to income, this is all about how to run the business better.

It is about identifying opportunities and acting on them.

It is about creating new opportunities and new demand.

It is sales, marketing, retention, product development, operations, risk, and reward.

It is the present and the future.

This diagram shows you what the commercial analytics landscape looks like and where it is developing. We will develop it step by step in this article.

Let us build it up one step at a time:

Execution

Commercial analytics starts with the execution because that is how we make money. In a marketing setting this means that you have a well-functioning campaign execution capability.

It is scientific marketing because it enables very fast test and measurement of campaigns. There should be no barriers to try out ideas and quantify them. You need this capability because you simply do not have all the answers and all the knowledge.

Typically, you should be able to get a campaign from idea to measured results (campaign velocity) in no more than two weeks for your standard channels.

Once you have tested and possibly refined your ideas into a winning, money-making campaign, then you execute it not just once but typically again and again. It may be the daily welcome programme, the cross-sell for the weekend activities, the fortnightly proactive churn activity, the monthly loyalty program, and the annual renewal. Whatever is right for your industry and business.

And the campaigns are not just triggered on time but triggered on events in the customer’s life, such as new purchases, changed usage, unusual behaviour, etc.

They are of course multi-channel and multi-stage campaigns where and email may be followed by a letter and then a call if the customer is of high enough value.

We do not forget that the best time to talk to a customer is usually when he or she contacts you. To do this effectively we need dialogue management or real-time inbound marketing, which is the decision support for our call centres, retail stores, and web sites that enables us to keep the conversation and the offers relevant and timely. Typically, for inbound marketing in a call centre you would present new offers to the customer 20-50% of the calls and expect a conversion rate of 25-50% of the presented.

Product, pricing, and proposition

Outside traditional direct marketing we do not forget product, pricing, proposition and go-to-market. We consider them to be execution channels for analytical insights. And like direct marketing we will be able to quantify the effect of our actions (though often with less rigour than then control group validated activities).

Insights should revels gaps in our product offerings as areas of our customers’ lives that we are not servicing (or not servicing well). Pricing strategies and bundles are another well-known outlet for insights, and is market research and product proposition and positioning development.

Customer View

The single customer view or the 360° customer view. It includes:

Customer attributes, which is your rarely-changing information like name and address and also socio-demographic information that you may have purchased. For example:

Name, address, postcode, email, …
Age, marital status, children, …
Payment methods and -details (credit cards, bank accounts, …)
Relations to other customers (family, business, …)
Contact preferences (marketing permissions)
Demographics (NRS social grade A/B/C1/C2/D/E, ONS socio-economic class 1-8, Mosaic UK Group/Type, …) – purchased or own

Transactions are the customer’s purchases, payments (including late and missed payments), and other information about how he transacts with your business. Examples:

Purchases (store, mail-order, online, …)
Bill payments (including late payment information)
Current and expired subscriptions (or other products / promotions)
Usage data (calls on mobile network, programmes watched on TV subscription, bytes transferred through ISP, …) if appropriate for the business

Interactions are when the business and the customer interact and are important to understand cost/profit (many calls to the call centre are expensive), as churn indicators (many visits to the ‘how to leave’ section on the web site) and much more. Examples:

Calls to the call centre, including call reasons and notes.
Visits to the web site, including what type of information did the customer look at (for example: adverts for new products or technical help for existing product)
Conversations in store.
Conversations with field agents (e.g. service or installation)
Indirect channel interactions (agents, distributers, field, etc.)

Attitudes: How does the customer feel? Traditionally through focus group samples but now increasingly using text mining and social network techniques.

Focus groups and market research results
Emails to the company (suggestions, support, sales, etc.)
Call centre notes are a rich source of customer sentiment attributable to individual customers
Comments on company forums, technical support sites etc.
Facebook, Twitter, blogs, and other social media outlets

Some of these are attributable to individual customers while others provides samples or aggregate information. All is valuable.

Campaign Data

Campaign data is information about which offers you have extended to each customer and how they responded.

Campaign information includes id, date, time, type of campaign, structure (e.g. multi-stage), etc.
Offer information includes information about the product, the promotion (e.g. 20% off), and the presentation (e.g. graphics, call centre script, ad copy, …), the channel, the agent, etc.
Outcome information includes click-through, shopping cart, purchase, usage, and any other relevant activity from the customer (e.g. ‘like’ the product on Facebook)

Data Hierarchy

Information will be available at different levels, depending on the source and the business context

B2C organisations will often have some information at the household level and some at the level of the individual.

B2B companies will often have a user – chooser type structure where one person or group (the chooser) may be responsible for buying for many users. (Think mobile phones, furniture, or computers.) And there will be complex company structures which may be important

Indirect businesses will have a further level in the hierarchy to include the distribution channel (e.g. agent or reseller)

Customer Analytical Record (CAR)

To simplify, many organizations introduce the Customer Analytical Record (CAR): a standard set of data, including aggregate data, stored as attributes against each customer over time. Typically there would be at least 24 months data and from a few thousand to ten thousand attributes. We aim that at least 80% of our analysis and reports can be done on this data set. Data may include

Standard customer attributes
Aggregated usage (last month, 3- months, 12 months)
Aggregated billing / purchases
Aggregated interactions per channel
Attitudes summary (e.g. satisfaction scores and trends)

Benefits of the Customer Analytical Record include:

Standard data set means consistent data definitions (single truth) and a gentler learning curve making new staff more productive faster.
Simpler format directly suited for many models and analysis (e.g. churn can be treated as a regression / classification problem rather than a time-series / hazard problem) reduces complexity and improves the time to execution.

Model Factory

The CAR enables us to have a Model Factory where the predictive models supporting our many ongoing customer activities and campaigns (see “Execution” above) are automatically refreshed and re-validated at regular intervals.

Markets and customers change, and so the behaviours of customers change. We want to avoid stale models that predict yesterday’s reality but does not deliver results today. We need to refresh the models to take into account current customer behaviour and we want to re-validate the models to make sure they are still predictive and stop any campaigns that are based on models with no (or negative) lift to allow us to re-analyse the situation.

People

Let’s talk about people. In our model, Analysts work primarily on the data in the CAR to produce dashboards, reports, and models. Models that are part of ongoing campaigns are included in the Model Factory for automatic updates.

Some analysts will have the ability to gather data from other systems, be they the main data warehouses or the operational source systems. We call them Data Wranglers and they are usually analysts with a longer tenure.

To complement Analysts and Data Wranglers you may have Team Leaders or other management function to handle scheduling, workload, quality assurance, and stakeholder management.

We sometimes borrow a technique from agile software development and use pair analysis where two people work together on the same workstation at the same time to develop the insights. One person, who is an analyst, types in the code or drives the analysis workbench. He or she is called the driver. The other person reviews the work and comes up with ideas. This person is called the observer and may be an analyst but we have found it extremely useful to include business people such as product or channel managers in this role, depending on the business problem.

It takes a little experience to know when to apply the pair analysis technique, but it can be extraordinarily rewarding.

Commercial, Customers, and Data

The single most important component for success of commercial analytics is this: to have people who all understand data, customers, and commercials.

Data is where the recruitment focus often lies: can this person manipulate data and create models?

Customers are crucial: the actions are actions for and with customers, actions to change customer behaviour. Team members need to be able to get into the head of the customer.

Commercial results is what it is all about and you can have great models and great marketing, but if the changed behaviour is not profitable or cannibalises other profits then there is really no point.

You probably will find it hard to recruit for this combination. It is relatively easy to recruit for the technical data skills. You can look at the candidate’s history to see how often it mentions money and customer behaviour; this will give you an indication. Mostly you have to grow this talent internally, and many of the techniques we have already described are designed to help you with this:

Start with the execution will help focus on changeable behaviours and commercials results. Pair analysis with observers from the business helps transfer commercial skills and customer knowledge. The CAR keeps you focused on the results, not the technical data wrangling.

The Analytical Shift

Times are changing. So far we have discussed traditional analytics which uses relatively static data (e.g. from a data warehouse), perhaps supplemented with key pieces of interaction data for real-time inbound marketing.

In this traditional world we tend to look for the norms: the trends, the clusters (e.g. segments), the networks, and so on.

Increasingly we are interested in exceptions: the deviations from trends, the outliers, and the network singletons or orphans.

This was always the case in the industrial setting where we were interested in machines that were about to fail or spotting defects.

The same approach applies to marketing and other commercial analytics. It is when I step out from the crowd that I reveal who I really am as a person. But this kind of processing requires different skills, tools, and techniques.

It uses bigger data because we are now analysing raw events, not just summaries. It is dynamic and real-time, and the results are expected in real-time. When I step out from the crowd is also often the right time to communicate.

And it is event stream processing which is very different from database processing: you now have to worry about sequence neutrality (same results if the events arrive in different order) and much more.

How we can help

Information: Web site, newsletter, and custom reports
- There is a wealth of resources and information available on our web site (www.cybaea.net): please use it and sign up for our newsletters.
- We understand this landscape and how it is evolving. We have been through the changes before.
- We provide custom research and reports using our industry contacts and knowledge.
Change: Rapid business benefits
- We can help you understand what you really need and what is optional.
- We can put money against the opportunities to develop a vision and business case for where you need to be.
- We can help you develop a change programme that is funded from the benefits it delivers along the way.
Talent: Interim management
- We can step in and provide talent you need to achieve success right now, and train or recruit for permanent operations.
- We only provide experienced people who have done this before and who deeply understand commercials, customers, and data. They are all credible at every level from the board to the technical analyst.
Service: Commercial Analytics as a Service
- For certain industries we can host the analytics and provide you with the norms, exceptions, and marketing actions.
- We work with a number of vendors of hosted software to provide the analytics capability as a service. For some organizations we provide a custom solution.
- We can provide the reports, dashboards, and insights or we can directly provide the marketing actions

Your thoughts

Please let us know your thoughts in the comments below or by contacting us.

Don’t forget that this material is also available as a stand-alone presentation:

Download CYBAEA Commercial Analytics Capabilities presentation (PDF).

Jump to comments.

5 common pitfalls of commercial analytics projects

2011-09-05T17:25:00Z

We have 20 years of experience of big data environments within a variety of industries including Research, Banking, Insurance, and Telecommunications. We have especially worked with customer data: Marketing, Risk Management, customer segmentation and -profitability, and customer-driven product development.

We have seen data mining and other analytics projects fail; we have seen insights teams unable to deliver the insights needed to actually improve the business; we have seen marketing teams unable to use data effectively to guide and quantify their activities; we have seen business leaders who are sitting on piles of data but are effectively flying blind because they can not get from the data to the knowledge they need to inform their decisions.

Below we have listed five common pitfalls of analytics in a commercial environment, their warning signs, and what you can do differently.

You may also be interested in our Reboot Analytics service, and we would love to hear your pitfalls in the comments below.

1. Not starting with the business actions

There are two pitfalls here. The first is that analytics can take on a life of its own and end up becoming its own purpose. For example, getting another nth decimal place improvement on the lift curve of your customer churn prediction model can easily become an obsession. However, it may not add much to the company’s profitability compared with better understanding the actions you can take that would change each customer’s propensity to leave and understanding what that action would do to your profits. (See also our longer article on Commercial Churn Modelling.)

Another example could be the search we sometimes see for “the one true” customer segmentation. We are not against a good and useful segmentation (see also our 5 step process for customer base segmentation), but where it goes wrong is when it becomes the only segmentation model and when all business has to wait for it to be completed, which often take a long time. But customers will not wait and neither will most business opportunities. Instead of strategic look at tactical segmentation models that address specific business opportunities, and do so quickly. There is not a single truth; only actionable insight.

The second pitfall is more subtle. Data mining with the objective of ‘finding something interesting’ is properly termed data exploration. It can occasionally be productive, but rarely so. A clear warning sign that the team has lost sight of the business is that they spend too much time exploring and not enough predicting.

It is a more subtle pitfall because you do actually want the data to help inspire your actions. But it has to be done quite systematically to make sure the process does not run away to take on a life of its own. You usually want a set of regular reports that are commercially meaningful and look at deviations from norms and trends. We have introduced our Insights Driven Campaign Creation process to our clients with big success. It is a process for systematically turning your data into insights and turning those insights into campaign ideas that generate real and substantial money. It takes the information from standard reports on customer Inflow, Base, Retention, and Outflow (IBRO), turns that into insights about the behavior of the customer base, and actions those insights through the whole marketing campaign process from concept to cash.

What to do?

So what should you do? Start with the actions in mind. What is it you can do that might make a difference? Where are your opportunities and threats? This is a business question, not a data question.

If you are worried about customers leaving you, you may decide that one thing you can do is to try to pay them to stay through loyalty programmes (think airlines) or simply discounts. If that is the only action you consider, then the questions of the data may be

What is the relationship between discount and change in loyalty?
What is the expected profit from the customer given a discount level?

In other words, if we give customer A a discount of $X, how much longer will he stay and how much more will he buy, and is that a profitable use of our capital? And what is this relationship for customer each of our customers?

Those are reasonable questions to try to answer with data about your customers. And they are quite different from the usual question of ”who will leave us?” They are questions about the effect of possible actions which what they should be.

2. Not considering all the actions

Another warning sign for your analytics project is that you are only considering the actions and activities that you have always done.

In the previous example, bribing your customers to stay may not be the only (or best) possible action you could take. Consider a mobile telephone operator that may hypothesise that customers leave them for four primary reasons:

Better call tariff elsewhere
Better handsets available elsewhere
Lack of coverage / signal (e.g. at home)
Other reasons

You could (and should) break it down further. For example, a better call tariff from a competitor may be because the customer’s friends are all on the other network in which case the possible action may be short term to offer a ‘friends and family’ type discount and medium term to entice the friends to your network, perhaps though a ‘refer a friend’ scheme. Or something else: these are just examples.

The point being that you create a (reasonably) long list of possible actions and you then investigate each one of them for profitability. The actions should be limited and specific: if possible do not offer general discounts but tailor the offer to the customer’s need.

Better call tariff elsewhere
- Create ‘friends and family’ plan
- Create ‘refer a friend’ plan
- Create international calling plan
- Promote soft benefits, e.g. exclusive access to events
- …
Better handsets available elsewhere
- Discount handsets in return for longer commitment (18, 24, … months)
- Explain benefits of handset range (Blackberry vs. iPhone vs. Android)
- …
Lack of coverage / signal (e.g. at home)
- Sell femtocells that provide local coverage using broadband
- Invest in infrastructure
- …
Other reasons

These are specific actions that recognise that customers are individuals and have individual needs, desires, and values. They are also actions that you can model and analyse from your data.

3. Not experimenting

You do not have all the answers. You do not understand everything about your customers. You need to experiment, experiment frequently, and not be afraid of (controlled) failure.

In Marketing there is a school of thought that periodically brings all the Big strategic managers into a Big room to think Big thoughts and come up with the one (or two or three) Big ideas that will save your business next year and really make a Big difference to your results. Because we know better.

Except we don’t. By and large we do not know better. Vision and strategy matters and you do need a sense of purpose and direction and a way of determining what you do not do. Ideas matter, and we are all for brainstorming and setting direction, but to think that you have all the answers is naive and to only focus on the few ‘big’ ideas is reckless.

Instead experiment. As long as you can control your execution and as long as you are rigorous in measuring (the right) results, experimentation is the way to learn. We built a whole civilization based on the scientific method and yet it is often ignored in a commercial setting by leaders who ‘know best’.

Fail early. Fail often. Enable yourself to execute many, frequent tests. Small, cheap campaigns to a limited number of people with careful data collection will get you a long way. You will be surprised by what you find.

4. Too much classification, not enough prediction

Ask your analytics team how much effort they spend on classification versus building prediction models. For most companies most of the time, the answer should be that almost all the effort is on the latter.

Are you drowning in the effort of producing ‘the one’ customer segmentation model? One company spent years and millions creating a single, global customer segmentation that was to be the basis for all Marketing and Product Development efforts. We ran the Marketing insights function for one of their countries and know that the only time we ever referred to this model was when writing reports to head office. It had no value to us in our work.

We see this pattern again and again. We are not against segmentation. On the contrary, we recommend that you have many segmentation models. Say, one for each /specific/ business problem or opportunity that you are facing at a specific point in time. Again: start with the actions and you will not get lost in the data.

We have written more about this topic at 5 step process for customer base segmentation

What to do

A useful segmentation should take days to create; a perfect one eternity. Focus your classification effort on specific opportunities, not general questions.

An analytics team that is spending too much time classifying is often one that has lost touch with the challenges that the business is actually facing. Instead the team is fishing the data to come up with “insights” to justify their existence (which is perfectly natural). Reconnect the business and the analytics. Have regular workshops considering the key challenges this week, month, or quarter, and what the data may be able to say that can help you understand and address them. Ideally they are interactive workshops with someone having live access to the data and a suitable query and visualisation tool so the questions can be answered and the ideas investigated then and there.

5. The ‘always more data’ and ‘data is IT’ traps

We need more data (or better data or cleaner data). Heard that one before? We have yet to meet a commercial organization that made everything they could with the data that was already to hand or which could easily be obtained (e.g. by talking to a few or yours or your competitors’ customers or ex-customers).

It doesn’t mean that data is not important, of course. But that if you are focused on what it is you want to do as a business then you should be able to be pragmatic and creative about getting data quickly.

Quickly being the key. We are (or should be) trying to exploit an opportunity or counter a threat in the market, so the last thing we want is a 3+ year IT project to create the perfect single view of the customer. Again, long IT projects may be in order, but they should provide infrastructure that enables the business to change quickly and not be the change itself.

Data is a business project, not an IT project. Data is there for a reason, it serves a purpose. Only the commercial business can define that and prioritise what is important. IT plays a vital role in enabling the business to do this, but don’t leave data to IT.

You need to make trade-offs in getting the data. Perfection is not required, commercial results are. If you leave it to a non-commercial function (IT) to provision data, then they can only attempt to deliver perfection, and that is doomed to fail. You want ‘good enough’ with the flexibility to change quickly for most things. If you are trying to market to your existing customers, then you probably do not need data to more than 10% accuracy on most dimensions. Except for the one or two dimension where you do need better data, of course. But it is not fair or productive to leave that to IT.

What to do

Do not let lack of data be an excuse for doing nothing. You can do more with what you have got and you can easily and inexpensively supplement it with useful, if imperfect, additional information that allows the business to move forward.

Do not let your IT function own the data. The business owns the data and only the business can prioritise data.

Do work with IT to build the data infrastructure you need. It needs to be flexible and pragmatic. It needs to be accessible to the business and the business needs to be able to manage it. That is a journey; a programme of several focused projects each of which delivers a valuable business capability will take you there, but needs strong and experienced leadership.

Do work with all customer-facing functions within the business to collect valuable data on your customers and their behaviours. Having the right data and having the ability to use that to guide and inspire your business decisions is one of the biggest sustainable competitive advantages you can build.

What do you think? Let us know in the comments.

Jump to comments.

Inflow segmentation – measuring new customers by value not volume

2011-01-06T20:35:00Z

Do you have accurate and timely analysis of the quality of the customers you are acquiring? Most companies carefully track the quantity of new customers by the hour, day, or certainly the week, but it is still less common to track the quality of the inflow as it happens. It is interesting to know that we have acquired, say, 1000 new customers today, but so very much more informative to know that this inflow will bring in £22,000 of revenues over the next year at 35% margin. Break it down by channel and product to see who is performing and who is not, and I as a marketing manager get really excited: I have the tools to do my job!

Monitoring the quality of the inflow and understanding the reasons for change is essential. After all, if your new customers are of lower quality than your existing base, then you are setting your company up for difficulties over many years to come.

Considering how much companies typically spend acquiring each new customer, this really should be a no-brainer. And yet many companies are completely unnecessarily stuck at reporting sales by volume instead of value.

We usually advocate a relatively simple segmentation of the inflow that allows you to track the quality in business-meaningful language. We want to do this segmentation very quickly after the acquisition but—at least for industries with a usage element as well as the base subscription—we also want to know something about the customer behaviour, not just the products. For mobile telecommunications customers we can usually segment the inflow about 6-8 weeks after the joining date into a dozen or so groups and predict each segment’s future annual spend to a few percentage points, which is fine for marketing and channel management purposes.

The segments will typically have meaning to the business. At one company we developed segments including Promo-Junkies (who just sign up for the initial promotion and never really use the product after the promo ends), DOAs (the dead-on-arrivals who sign up but never use the product), Champions (the high spenders), and so on.

We can track the inflow segments over time. The slightly annoying animation in the margin shows 15 months of inflow segments. Here the size of the bubble is the number of customers acquired, and we have projected the segments to two dimensions: a spend metric along the abscissa and a usage metric along the ordinate. (Yes, that really is a whole lot of customers at zero spend which may be the subject of another article.)

We clearly see an important seasonal effect which tells you a lot about your business and, in this case, especially about the effect of your seasonal promotions. For this particular client we saw that the Christmas promotion substantially increased the inflow of Promo-Junkies and DOAs, as the sceptics had predicted, but also both the number and the spend of the Champion segment which amply justified the promotion.

For another customer we saw that the quality of customers acquired by a particular reseller channel was very low, both compared with other channels and in absolute terms: these customers were not profitable. We tried hard to work with the channel but, when that ultimately failed to bring about change, we dropped our relationship with what was one of the largest independent reseller of mobile phone contracts in the country, significantly increasing the overall profits of the business while reducing revenues.

If the quality of new customers is dropping, then you want to know about it quickly to determine if you have a problem and where the issue lies. Maybe you have an under-preforming channel or a new product or marketing message are driving a different customer mix to you. Conversely, if the quality increases you want to understand why so you can encourage more of that behaviour.

Other industries will be different in the details, but if they have data then they can apply the same approach. For the Insurance business it depends a lot on the market and how much information you get about the customer when he joins, but there you can usually develop good models for very shortly after the joining date if you can get Marketing and Underwriting to work well together (no mean feat). For Retail Banking we usually need 2-3 months until all the direct debits and standing orders have moved across and we have seen a couple of pay-cheques.

If you have customers and you have a regular usage and subscription element to your relationship with them, then you can and should use inflow segmentation to monitor the value of your customers, not just their volume.

Let us know how you manage the value of your new customers in the comments below.

Jump to comments.

Understanding reasons for churn – and what you can do about it

2011-01-05T12:01:00Z

We argued in our article on commercial churn modelling that you want to predict not only the probability of a customer leaving you but even more importantly what you can do about it. We want to predict why the customer is churning or, more precisely, his likelihood to stay (given that he was likely to leave) after we extend an offer or perform an action from a list of activities for churn management, as well as his profitability after the save.

In the previous piece we did not consider the question of how you determine these reasons for churn, so let us turn to that briefly here.

You could try asking the customers who are leaving. This is unlikely to give you the answer you are looking for, but I still recommend that you do it, and that you do it regularly.

The idea here is to ask people who have just left you why and what (if anything) you could have done to keep them. Make sure that this is not a sales call. Try a script in the spirit of: “I know you have just left us and I am not trying to change your mind—I just want to ask you what made you leave and what we would have had to have done to keep you: if helps us to become better.” Do this for a sample of your leavers every week or month and track the answers you are getting.

If you are like every other company I have ever done this with, then by far the majority of people will answer “price”, which is why I said it would be unlikely to give you the answer you need for your commercial churn modelling.

But it is still a very useful exercise. First, understand the answers that were not about price, track them as they change over time when your competitive landscape changes, and devise saves activities that profitably address these issues.

Second, dig into those “price” answers. Is it really about the core price of your product offering, or something peripheral which you could change without incurring wholesale margin erosion and ideally change below-the-line only? Maybe you are in the insurance industry and people leave because a competitor has a good offer for bundling all policies with them, in which case you can address this specific price without touching individual policies. Maybe you are in the mobile telecommunications industry and a customer is leaving because he gets cheap on-net calls with a competitor: can you get his friends and family onto your network and save him that way?

Keep talking to your customers and keep trying to understand what are precisely the offers or activities that will retain them, rather than just reaching for across-the-board price cuts.

Customers think or say that price determines their actions even though that is not the case nearly as often as you might think.

Learn by doing

So if asking the (recently departed) customer is not going to give you the answers, what will? As usual in marketing, you need to test, measure, and learn.

You have some sort of saves activity going on for your customers. (You have, haven’t you?) Assign and extend a random retention offer to each of a random selection of your churning customers. That gives you the answer to what you really want to know: what offers work in retaining which customers. Very quickly you will begin to discern the clusters of customers that respond to the different offers. Ideally, you will have real-time predictive analytics installed in the front office to continually optimise the offer mix (and most solutions can do well on this type of problem) or any half-decent analytics team should be able to run the analysis daily or weekly and update the offer mix and rules.

The beauty is that as you keep running these trials you get more data with allows you to see deeper into your customer base and identify smaller behavioural segments. You can also test many more offers over time, helping you to manage your margins ever better and ever smarter than the competition.

(You will eventually have to consider the people you do not reach at the time they churn to understand their behaviours and feed that into your product and advertising plans, but let’s fix the ones we can save first.)

What are your best ideas for churn management? Let us know in the comments below.

Jump to comments.

Commercial churn modelling

2011-01-04T11:29:00Z

Churn modelling is easy; commercial churn modelling is hard. Let us compare the two to explain what we mean by the latter.

Consider any subscription or subscription-like business such as telephones, broadband, television, credit cards, insurance, retail banking, mail order or any other business with substantial repeat purchases. If you run any business like this then you will be overrun with companies offering you churn modelling and promising that they have the secret sauce that will make the model a little better than any other model and save you millions or billions.

(We consultants like to sell churn models because the business case for a one percentage point improvement in churn reduction is usually an astronomical sum that dwarfs anything else you may be doing.)

But here are the dirty little secrets of the churn modelling business that you should know:

Traditional churn modelling is easy
Having a few extra decimals on the prediction of a traditional churn model will only make a meaningful difference to your business if you are in land-grab mode in a price-led market.

On the first point, you can create a churn model at over 75% effectiveness (which for my technical readers I define as the area under the sensitivity–specificity curve) as a purely mathematical exercise without knowing anything about your business data. The KDD Cup Challenge 2009 shows this for a data set from the telecommunications company Orange where all the data attributes have been hidden. There are at the time of writing over 200 submissions with a churn score over 75%, and one of the original winners used only a couple of laptops to achieve this.

We do not use laptops but scalable clusters of computers for our analysis, but we can show similar results for several other industries.

Traditional churn modelling is really easy.

And if you apply your understanding of the business, customers, and markets to the problem (as you would outside an artificial competition like the KDD Cup mentioned), then you will of course get even better results, with 80-85+% not uncommon in true subscriptions businesses like telecommunications.

Again: Traditional churn modelling is easy.

It is also mostly useless from a commercial perspective.

What are you going to do if you think someone is about to leave in the next month/quarter/year, other than to offer him a discount?

If all you know is that he will leave, all you can really do is to pay him to stay. Margin erosion, commoditisation of your products, and much gnashing of teeth is certain to follow if you adopt this as your churn strategy. If you are trying to buy market share against less well-funded competitors, this may be a rational approach. But in most circumstance is it something you want to minimise and manage carefully (few industries can avoid it completely).

It is true that in industries like mobile telecommunications the marginal cost of adding one more subscriber to the network is near zero and Economics 101 predicts that this will eventually be the price of the product. However, as business people we normally want to fight this slide every single step of the way to maintain margins, recoup large investments (in mobile networks, say), and fund future developments.

Commercial churn modelling

We call our approach commercial churn modelling, which may not be a great name (suggestions for better names in the comments, please) but at least it tries to bring the exercise back to where it needs to be: actionable, and actionable in a way that delivers commercial results.

If a customer is about to leave and there is nothing at all I can do about it, nothing I could offer or do to make him stay, then I don’t really need to know about it. Knowing may be useful for budgeting and forecasting, but that is not how we make the big money or the big impact in society. (Accountants who might disagree can comment below.)

We don’t really want to know that a customer may leave, we want to know what would make him stay. The business challenge is then to deliver that incentive (or a substitute) in a profitable way.

Consider a mobile telecommunications company. We could hypothesise that the four main reasons for a customer leaving were price, handset, coverage, and network (data) speed. Knowing that a customer is likely to leave and the reason why, is actionable. If we are genuinely more expensive than the competition, I could offer a discount or explain the extra benefits of staying with us. If we do not have the latest handsets, I could offer an even better one that we do have at a competitive price. At one company we worked for, I even went out and bought an unlocked iPhone (which we didn’t offer) and shipped it personally to one very loyal and profitable customer. Coverage is a little harder to do something about, but maybe I could offer a femtocell that uses the customer’s own broadband to provide coverage, as Vodafone is currently doing, and for the slow data network I was able to offer nationwide WiFi hotspot access when at the company where we sent the iPhone.

The point being that knowing the likelihood to churn and the (most probable) reason for leaving is actionable by the business in a way that knowing only the first component will never be. Only one of the reasons above is eroding my core pricing and even there I can dig further into why there is a price gap and maybe do something about that instead of a unthinking margin reduction. (For example, the customer’s friends may be on another network and the cheaper on-net calls are the reasons for the price differential which is certainly something I can proactively prevent through social network analysis and reactively try to change through refer-a-friend or similar programmes.)

This is what we do here at CYBAEA and what we mean by commercial churn modelling: predicting not just that a given customer is about to leave, but what you can do about it right now. We additionally develop analytics that predict changes in the customer’s behaviour after accepting an offer, and therefore the change in revenues and profitability, which is what you need to make a rational commercial decision about what to do with each customer at each moment in time.

The reasons will vary between industries as will the technical details about how you model them (independent or conditional models, for example), but the approach is applicable everywhere you have a subscription or subscription business, and the results have the potential to transform the profitability of the business.

So just say no to the traditional churn modelling and make sure your provider really understands your commercial environment. Ask hard questions. How will I use this model in the business? What will I actually and specifically do with the output?

Jump to comments.

Why?

2011-01-03T10:10:00Z

Why do we do analytics? You will come to know the truth, and the truth will set you free, said the teacher, and while he wasn’t talking about commercial data mining we think he could have been.

The freedom here is not the freedom to do nothing or to do anything, but the freedom to do the right and proper things, and to do them well. Happiness comes not from idleness.

But how do we get the wisdom to know what is the proper pursuit for us? That wisdom can come from knowledge, and the knowledge from information, as the poet writes:

Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?

This inspires us here. To help you recover and rediscover the knowledge that is hidden in your data; especially knowledge and genuine understanding of your customers and markets. Knowledge of people, for it is in and through the community that we live; and from that knowledge we may come to understand ourselves and our role in the world, which is true wisdom as the Delphic oracle well knew (γνῶθι σεαυτόν).

So we developed our Insights Driven Campaign Creation process to systematically take insights from reporting and data mining and turn them into new testable marketing campaigns, month after month, year after year. For it is through actions and communications that we understand the humans that are our customers and markets.

And we developed the very commercially focused dashboards, reports, and fast analytics that can feed this process, and the business structures, organisation, and leadership that commercially optimises the results.

Some examples of what we do
Customer Base Management	Insights Driven Campaign Creation	Commercial Marketing Dashboards and Analytics
Organisational design & development Design and implement P&L for Customer Base Management Interim management	From insights to campaigns, month after month Campaign test–measure–refine–roll-out execution Commercial campaign effectiveness reporting	Inflow–Base–Retention–Outflow (IBRO) reporting Customer inflow (acquisition) segmentation Social network analysis (SNA) Commercial churn modelling

We wish all of our readers a happy New Year.

Jump to comments.

Employee productivity revisited

2010-06-22T11:45:00Z

We have a mild obsession with employee productivity and how that declines as companies get bigger. We have previously found that when you treble the number of workers, you halve their individual productivity which is scary.

We now re-do the analysis four years later and, just because we can, we are using the leading companies of the London stock exchange instead of the largest American companies.

The results also show in this data set. We called it the 3/2 rule: treble the number of workers and you halve their individual productivity. Large FTSE-100 companies with ten times the number of employees are ¼ as productive as their smaller competitors and if all the FTSE-100 companies achieved their average profits per employee, then the index would generate almost £1 trn of additional net profits for the economy. (But see the comment about the likely selection bias before you rush out to de-merge your company.)

So what is happening? We can postulate a couple of effects. Companies may legitimately choose to be in a high-volume, low-margin business. High-margin business may not scale well or scale with the number of employees. And at least for the FTSE-100 analysis, choosing the biggest companies excludes any with both low number of employees and low revenues per employee (see also The 3/2 rule revisited).

But I wonder how much of this effect is down to communication. Bigger companies may have more meetings with more people attending slowing down decision times, innovation velocity, and productivity. It sounds right for many of us who have worked in companies of different sizes.

You can see the technical details of the analysis and run it yourself at: Employee productivity as function of number of workers revisited.

Jump to comments.

Your mobile phone knows everything about you ... and it is telling

2009-08-17T09:18:00Z

We knew the potential existed already, of course. Mobile devices in the USA generates some 600 billion transactions per day, each tagged with the location and time. Jeff Jonas: Every call, text message, email and data transfer handled by your mobile device creates a transaction with your space-time coordinate[...]. Got a Blackberry? Every few minutes, it sends a heartbeat, creating a transaction whether you are using the phone or not. That is some 7 million transactions per second, on average.

The mobile operators have this data, of course. We all know this (especially here where we have been using some of it for social network analysis), at least in principle. No real surprises here, except perhaps in the volumes.

But did you know that the operators are sharing your data? What is new, at least to me, is that this data is being provided to third parties that are leveraging specially designed analytics to make sense of our space-time-travel data. With the data out and specialized analytics emerging, this infant industry is already doing some pretty amazing work. Your space-time-travel data makes where you live and where you work self-evident, and it reveals your most frequent, periodic, infrequent and rare destinations.

This is powerful stuff. I can't validate Jeff's claim that the network operators are sharing their detailed network data in any systematic way and for anything other than research purposes, but surely it is only a matter of time. This data is too good to ignore — super-food for analytics, as Jeff calls it.

Think about the possibilities. We obviously know where you live and where you work, but also who is in the house with you (your wife, hopefully) and which of your office mates join you for the Friday afternoon beer.

We know which department stores and malls you visit. We know where in the stores you spend the time. We know your hobbies, your interests, your friends, .... We know your mistress and your bank manager, your colleagues and the boys on the football team you coach. We know where you shop and when you shop, we know your commuting route and that you always sop at Starbucks for a coffee on the way in. We know it in real-time so we know you are going to be late in the office this morning. Would you like us to let them know that you will be there at 09:23 (or 09:12 if you do not stop for that coffee on the way)?

There are so many possibilities. Much good could be done with this data. Some bad. Which service would you like to see developed?

B2B Content Marketing

2009-07-22T06:59:00Z

The nice people at Velocity has released The B2B Content Marketing Workbook. It is behind a registration wall which means we wouldn’t normally recommend it but you can just type junk in the fields if you are not comfortable with giving your personal details to a marketing agency. (Think about it....) If you are relatively new in the B2B world, say having joined a professional services or consulting organization, you may find this one useful.

The basic message is that content is king. In B2B you are selling your expertise. The author’s main expertise is with technology companies and the statement bears repeating in that context. You are not selling a product. You are selling expertise. You are the best at what you do, you have the best people in the industry working on the problem you are solving, and it is this expertise you are selling to your business customers. The product is just a handy way of packaging up some of your knowledge and experience, but by no means the only one.

Marketing in B2B therefore means content, content, and more content. The book covers various forms of content from the traditional white papers to video blogging with links to examples (from Velicity, naturally). Have a quick scan: there may be something useful for you.

PS. A quick tip: there are some technical problems with the PDF file and if you have problems reading some of the slides try to select all the text which will allow you to read what is rendered as white text on white background.

Marketing lessons from Antiquity

2009-07-10T21:25:00Z

A story from antiquity involving a king of Rome and a Greek Sibyl has lovely marketing lessons.

Sometime around 576 BC the Cumaean Sibyl arrives in Rome and offer nine books of her prophesies to King Tarquin, the legendary (in both senses of the word) last king of Rome (they had emperors after that).

The king laughs at the enormous sum she is asking for the books and sends her packing. She then burns three of the books and goes back to the king offering the six books at the same high price as the original nine. He rejects her again.

Then she burns three more of the books and offer the last three to the king at the same price.

This time he buys them.

The Sibyl was a good marketing person. She understood scarcity in volume. Three books can be much more valuable than nine.

She understood scarcity in time. If you don’t act now the opportunity is forever gone.

Interestingly, she didn’t try to auction the books to the highest bidder. Ignoring the logistical difficulties of doing that at the time (she was in Naples), it probably wouldn’t have worked. The scarcity in time pressure is so much greater when the consequence of not buying is that the opportunity is lost to all mankind. It isn’t like you can regret your non-purchase and acquire the books later. The Sibyl was very smart.

As a historical aside, the three books of prophesies were finally burnt in AD 405 by General Flavius Stilicho who as a committed Christian considered the pagan books evil.

Does social networks influence purchases?

2009-05-22T02:35:00Z

Havard Business School has an interesting study titled Do Friends Influence Purchases in a Social Network?. I would like to get my hands on the raw data (which is from the Korean social site Cyworld), but the outline conclusions seems plausible:

Highly connected people are negatively influenced by purchases in their network. This is consistent with a hypothesis that they are trend leaders who are always looking for new things to differentiate themselves from their group.
Moderately connected people are positively influenced by purchases in their network. This is consistent with a hypothesis that these people are trying to "keep up with the Joneses".
Weakly connected people (48% of the data set) are not influenced by purchases in their network. The paper rather offhandedly dismisses these people as "the low status group".

Interesting study and worth considering the implications for your social network analysis. As an aside, I was talking yesterday to a Marketing software company who were exactly focusing on that middle segment in their default parameters for their SNA module. Clever.

KDD Cup 2009

2009-05-12T10:02:00Z

The results from the KDD Cup 2009 are both interesting and fundamentally not interesting. For this public data mining challenge Orange, the mobile telecommunications company, provided anonymous data sets on mobile customers: 50,000 records each of training and testing data with 15,000 variables. (The data set are still available for download and there are also smaller data sets with only 230 variables.) The competition was to provide the best models for churn, cross-sell (“appetency”), and up-sell.

The problem with the competition is that we do not know what the data means: the variables are simply named Var1, Var2, ..., Var15000. This means that this is purely a statistical exercise and no understanding of the business problem is required or helpful. Which is really disappointing and made the challenge much (much) less interesting for me.

What is kind-of-interesting about the results is that you can still score some 75% of the churners (where the score is the area under the specificity-sensitivity curve). That is a little higher than I expected from purely statistical methods (my guess would have been around ⅔).

The message to mobile operators is that if you do not know (about) 75% of your churners in advance, you are doing something very wrong. You are not even getting the statistics right. And if you know your business you can relatively easily get the score up in the 80-90% range: we have done that without getting very sophisticated in the analysis. Predicting who will churn in mobile telecommunications is not hard.

However, the number isn’t really interesting. Let’s just assume the sensitivity is 75% so you know that proportion of churners in advance. Or even 80% or 90%. So what? Prediction is not the goal in business, action is. First I need to know who to retain. If the 25% (20% or 10%) I do not predict correctly are all the profitable customers who are churning, then I am no better off than without the model.

And second I need to know how to retain them. Profitably. That means understanding why they are churning, what offers would potentially retain them, and how they will behave in the next contract period so I can see if it would be profitable for me to extend the retention offers.

That is hard. The problem is not predicting who will churn. And not even to predict which profitable customers will churn (the first objection above), though that is somewhat harder. Retention is not difficult. Profitable retention is. Predicting which offers will retain the customer and how they will behave given that offer is hard. And that is the (only) interesting business problem in retention.

Jump to comments.

The financial crisis and physicists

2009-03-20T10:04:00Z

The financial crisis is all my fault. Or so David Smith from our friends REvolution seems to suggest in his post Physicists, models, and the credit crisis:

I remember working in The City in the late 90's and Wall Street in the early 00's and remarking then that just about every quant had a physics or engineering background. I met very few statisticians. Quantitative models have taken a hefty share of the blame for the credit crisis, but I wonder whether the blame lies more in their application, rather than the models themselves. Statisticians are trained on the limitations of models, and how to detect when models are breaking down, but statisticians were woefully underrepresented amongst quants. Do physicists and engineers get similar training?

I can only speak as a (ex-)physicist (I used to work on experiments at CERN) and the short answer is "yes". Of course we do get this training.

I did experimental physics. What we are good at is testing a model (or a set of competing models) against data. Physics only really moves forward when (1) we find some domain where our previous models do not work or (2) we find a simpler model that consolidates several previously separate models or extends the domain of a previous model.

The quantum properties of particles would be an example of the first and general relativity might be an example of the latter.

The caveat is that it really helps if it is a "reasonable" model we are looking at. We are not primarily trained in statistics. We are primarily trained in understanding nature: cause and effect, the scientific method, and a mathematical apparatus for communication.

A statistician is primarily trained in statistics. He will run tests on the model against data. I firmly believe that no statistician should ever be allowed out in the wild unsupervised. A physicist is trained in a world-view, if you like. He will (first) try to understand the model. He will look for internal consistency and he will compare with other models in the same or adjacent domains. He will run thought experiments. Then he might run some statistical tests against data.

I worked as a quant in a bank back in the days before "banker" became a four-letter word. We did good models. But, and this is important, the focus was on modeling the high-volume trades better. That's where the money is. If you could find an anomaly there you were set to make millions. If you could find a better model there you could make hundreds of millions.

If you could find an anomaly on the tail, you could write a research paper.

The average tenure of the people I was working with was less then three years. Nobody was interested in events that were not likely to occur within a three year time horizon. We knew and understood that the models were not valid on the tails, but there was no volume of trading on the tails so it wasn’t very interesting.

We knew that the tails didn't fit. But that was not important. Making money is the business of business.

Quoting from the article that got David Smith started:

Quants say that they should not be blamed for the actions of traders. They say they have been in the forefront of pointing out the shortcomings of modern economics.
“I regard quants to be the good guys,” said Eric R. Weinstein, a mathematical physicist who runs the Natron Group, a hedge fund in Manhattan. “We did try to warn people,” he said. “This is a crisis caused by business decisions. This isn’t the result of pointy-headed guys from fancy schools who didn’t understand volatility or correlation.”

Business decisions. But before we go completely medieval on the bankers, let us understand that society is often expected to pick up the bill for tail effects. Houses built in flood plains, hurricane prone area, or earthquake zones? DDT? Space debris? Global warming? Even famines are probably mostly preventable, but at a cost.

And the challenge with with all of this is that the cost of prevention may be bigger than the cost of fixing it. Maybe. It is hard to say: both costs are typically highly uncertain, and in the (really) long run we are all dead and extinct anyhow: if you take the long view nothing is cost-justified. This universe will end.

The question is: for all the billions of (preventable) damage caused by the current crisis and for all the personal losses (everyone without a job) and tragedies (everyone without a pension or conned by Maldoff), did the banks create net value summed over both the good and bad years? Capitalism has allowed us to grow albeit in fits and starts. The bankers made huge sums of money in the good years and not just for themselves but real money in real business and real growth and innovation. Ask the people who would not have been alive today if it was worth it: the poor who got additional aid and support (often squandered by their leaders but still) from the surplus wealth generated by all the elements of capitalism, including those bankers, and ask the people benefiting from new medicines developed by capital-intensive pharmaceutical research. Ask the people benefiting from capital investments in infrastructure projects: hospitals, hydroelectric dams, schools, etc.

We have the luxury to be angry with the bankers. I do not want to take that luxury away from us: it is wonderful that we can afford to indulge ourselves. But I would like to remind us that is is a luxury that we can afford, and that we can afford it at least in part due to those bankers.

Jump to comments.

Happy birthday WWW

2009-03-12T22:34:00Z

Twenty years ago, on 13th March 1989, Tim Berners-Lee wrote the original proposal for what was to become the World Wide Web. Happy birthday!

I worked at CERN from around 1990 to 1994 in an office just across the road from Tim. I still remember going to one of his early presentations and afterward saying that if is a nice idea but “it will never take off.” If you read this Tim: sorry: you were right and I was wrong (but perhaps you shouldn’t have presented it as a way of doing collaborative detector design: I still don’t think we have enough bandwidth for collaborative CAD/CAM but I left CERN in 1994 so maybe things have evolved).

Tributes:

5 step process for customer base segmentation

2009-02-26T11:15:00Z

5 step process for customer base segmentation

Over the last years we have been doing a tremendous amount of customer segmentation work with the marketing departments in companies across a number of industries. We have experienced that there are many misconceptions about what “segmentation” really is, why we do it, and what we can expect to achieve from it.

All too often marketing departments thinks that database analysis is the first, last, and only step in segmenting the base of existing customers. In fact, identifying clusters of common behaviors is only the first activity you should undertake in creating a customer base segmentation.

Customer segmentation is not a piece of database work. It is a strategic or tactical business activity with with hard monetary benefits. Yes, you do need to do some data mining, and how clever you are in doing that is important, but that is not (or should not be) the primary activity.

Assuming that you are doing segmentation for the right reasons and you therefore know how to measure if you are successful, then what are the steps you need to make it happen?

We have found the following list useful:

Database analysis
- Behavioral clusters
Market survey
- Behavioral segment descriptions
Market strategy
- Brand and market positioning
Market research
- Strategic segments
Needs-based segmentation
- Development segments

This is not intended to suggest that you must always perform all the activities on the list. Sometimes you may only need some of them to achieve your objectives. But it is an ordered list so start at the top and work your way through until you have achieved (and measured!) your goals.

Let’s look at the steps in turn.

1. Database analysis to determine behavioral clusters

The first step is to take your customer data and analyze it to determine clusters of similar behavior. Much has been written on the subject elsewhere so I will not go into technical details here.

But from a business perspective it is essential that you know why you are segmenting so you know which variables you want the clusters to split. We discussed this in some detail in our previous article about the 3 things we want from customer segmentation.

You want the segments to be different on the variables that are important for your business. Otherwise the segments are not useful. All too often we see marketing departments commissioning data mining without specifying clearly how they want to use the resulting segments. “Find us some clusters” is not a project brief. The analysts will find you clusters without any problems, but they will not be commercially significant or deliver the bottom-line benefits that you need.

2. Survey your customers to obtain behavioral segment descriptions

Sometimes the database clusters may be sufficient. Maybe you only want to improve the targeting of some existing campaigns. But if you want to consider developing new campaigns or new propositions then you need at least some business understanding of these customers. Not just what they do (step 1) but also who they are (step 2).

Usually you will to this by asking them. You take a small random sample and interview them about who they are. This is classic market research and there is plenty of available literature on the subject. You know what you need to do, just realize that your organization is probably not very good at it unless it is a specialized agency and it has a large budget (which is rarely cost-justified). But it doesn’t have to be perfect, it has to be useful. You can (and should!) almost always trial your campaigns first to see if they really deliver what you expect.

3. Determine your market strategy and your company’s position in the market

You’ll want to understand your organization’s desired position in the market so you can develop the right segments witht he right propositions. It is that vision thing again.

You need to have a vision for your company and where it is going. As a marketing manager, it is your responsibility to translate this into customer segments. Who are our customers. Who are our non-customers? Who are our customers that are not using all of our services or products? How do we target them better and how do we articulate the value that we as a company can add to their lives.

You need this when you want to develop new campaigns, new marketing messages, and new propositions.

4. Determine your strategic segments

This is segmentation as a strategic business tool. You’ll want to understand your organization’s desired position in the market so you can determine which segments to develop. Depending on how strategic the marketing department is considered with the company, and depending on the status and ambition of the marketing manager, this may be a step too far. But if you want to play at the top table this is what you do.

One company we worked with commissioned additional market research on their segments to measure them on two dimensions: lifetime value (x-axis) and how well the segment was aligned with the organization’s strategic direction as expressed in its vision (y-axis).

They then focused their attention on the segments to the right of the dotted line. That does not mean giving up on the others, but most of new campaigns and propositions target the strategic focus segments.

This picture proved to be an important tool in communicating at the CXO level what the company was about, where it was headed, what the challenges were, and where the opportunity lay. This is an extremely powerful tool.

5. Needs-based segmentation for proposition development

There may not be enough growth opportunity in your strategic segments from the previous analysis. Or maybe you are responsible for developing new propositions. In that case you want to develop an needs-based segmentation not just of your customers but of the whole market.

You can start with your customers and understand their needs, especially the needs your company is not currently fulfilling. In fact, you should start by analyzing and questioning your customers since (a) it is much easier to sell to an existing customer and (2) you (probably) know who they are so they are easy to contact.

Eventually you will probably want to commission market research to identify unmet needs in the general population of potential customers. But this is expensive research and it is hard to make sure you develop the right offers and very expensive to bring them to market. So make sure you get the most of your existing customers.

Organizational alignment to make use of segments

We will talk more about this some other time. But as the marketing manager it is your responsibility to clearly demonstrate the business value of the segmentation that you are developing at each step in the process and to ensure organizational buy-in to use them as the reference for communication and activities. Segments are not just for marketing.

Jump to comments.

3 things we want from a segmentation of the customer base

2009-02-26T08:16:00Z

3 things we want from a segmentation of the customer base

Why are we doing customer segmentation? When we come in there is obviously a desire to “make more money”, and that is not a bad top-level ambition. But to translate that into specific set of activities you need take your thinking to the next level of understanding. And this is something we find that most organizations have failed to do.

Here is a list that we have found useful as a starter and which you can use as a base for your own work:

Meaningful segment descriptions
- This will help us with proposition and campaign development
- Increase the proposition profitability
- Increase the number and quality of campaign ideas
Stable segments for stable customer bases
- The segmentation should tell use something about our customers and not just about externalities
The segments differentiate on the variables that drive our marketing campaigns
- If X is important to our business then not all segments should have the same distribution of X; in fact the distributions for different segments should be as different as possible
  - X could be profit, life-time value, usage, churn propensity, channel preferences, ….

The important point about these is that they are all measurable. If we can measure it then we can build something that delivers it. Let’s look at each briefly in turn.

Meaningful segment descriptions

We want to use the segments in the business, so knowing that the segment is the fifth to eighth decile of the third principal component or some such is really not helpful. Knowing that it consists of plumbers and the like who work for themselves or in small teams and use their mobile telephones primarily on the job to keep in touch with their customers may be useful.

We will know that it is useful if it helps us develop new products, offers, or services (propositions) and if these are profitable and advance our business goals (increased market penetration, margin contribution, or whatever they may be).

The segmentation should help us to communicate more effectively with our customers. That usually means talking to them about the things we offer that may genuinely be interesting to them, and only about those things. Communication and dialog instead of spamming. We need more marketing campaigns and therefore campaign ideas, but these campaigns are smaller, better targeted, and deliver much higher returns.

We have introduced our Insights Driven Campaign Creation process to our clients with tremendous success. Insights Driven Campaign Creation is a process for systematically turning your data into insights and turning those insights into campaign ideas that generate real money. It takes the information from standard reports on customer Inflow, Base, Retention, and Outflow (IBRO), turns that into insights about the behavior of the customer base, and actions those insights through the whole marketing campaign process from concept to cash.

Stable segments for stable customer base

This should be a rather trivial point but in our experience it is not. Your segmentation should tell you something about your customers, and not just externalities.

We looked at a segmentation done by a very famous data analysis company. We were concerned about the methods and variables they used to determine the segments so we investigated their stability. We ran the segmentation for three consecutive 3-month periods and we looked to see how many were in the same segment from one period to the next. We knew that at least our high-value customers had been with us for long periods of time (more than 36 months) and so we expected that most users (say, much more than 50%) would be in the same segment from one quarter to the next.

What we found was that typically ⅓ of the customers in a segment fell into the same segment the next quarter. That is a truly awful segmentation for this client with a large, stable, long tenured customer base! Looking at the detailed technical definition of the segments it was not hard to see why it was so bad: it was extremely sensitive to a few variable with a very irregular usage and a usage that was heavily influenced by our marketing campaigns. So the segmentation primarily told us if our customers had been the target of our own marketing!

The segments differentiate on the variables that drive our marketing campaigns

Another point that should be trivial but most emphatically is not in many of the organizations we encounter. We find these pre-existing segmentations with fancy titles and superficially engaging descriptions, but when we look at the things that actually matter to the business we find no significant differences between the segments!

Remember that you actually want to do something with your segments. If your company offers subscription based services and all segments have the same churn profile then the segmentation is not really going to help you manage your churn problem. If all segments have the same usage profile, how are you going to use the segmentation to stimulate usage and increase revenues? If all segments have the same distribution of preferences for receiving communications by post, telephone, and email, how will the segmentation help you manage your marketing communications better?

This is probably the subject of another detailed post to consider when you should use what kind of segmentation approach, but for now at least consider the things that are important to the operations of your business in the next 6-12 months and see if your segments actually have very significantly different distributions on these attributes.

If churn is important to you, then you want loyal and opportunistic customers in different segments. You want to be able to look at what customers are valuing so you can offer them the best incentive to stay (and develop a compelling proposition for them if you do not already have one). So you probably should have people who value price more than service in a different segment from those whose prioritize convenience over costs. You undoubtedly want different profitability in different segments so you can manage then differently (e.g. move low-margin customers to self-service channels). And so forth: the details will vary a little with your industry but you get the idea. Do not assume that your segments have different distributions of the things you care about: measure it.

Jump to comments.

The truth about venture capitalists according to Marc Andreessen

2007-06-10T14:00:00Z

Marc Andreessen, who as founder of Netscape and other companies knows a thing or two about the subject, has a nice little series on The Truth About Venture Capitalists, part 1, part 2, and part 3. We've said something similar before, but Marc puts it very well.

Also check out his productivity guide. Gotta love the first tip: don't keep a schedule: refuse to commit to meetings, appointments, or activities at any set time in any future day. Radical, certainly, but interesting.

Innovation forever

2007-05-27T19:38:00Z

We are keen on innovation here at CYBAEA, so I feel obliged to mention two articles on the subject that I noticed this week.

The innovation horizon

The New Scientist (whose web site today crashes on connections from the UK but works fine when connecting from the US) reports (subscription required) that innovation is the driver for growth in cities, and that innovation grows as a power law against the number of inhabitants in the city with a factor of 1.2. This factor is greater than one, which has two interesting consequences.

First it implies that there are no upper limits on city sizes. The largest megacity is Tokyo with 35 million people, followed by a handful of cities around the 20 million mark. There is no reason why these urban structures should not continue to grow in size. The only limit is the limit on human ingenuity and innovation, since that is the driver for growth.

From New Scientist

Second, and slightly contradictory, the models suggests that we are approaching a kind of innovation horizon where infinite innovation in infinite short time is required to sustain human civilization. The structure of growth is on of an super-linear boom followed by a "bust" which resets the growth to one with a slightly longer time to infinity.

But only slightly longer. If you reset superlinear growth at the time of a bust, the duration until the population would grow infinite again gets shorter. This agrees with our experience in innovation: new ideas and technologies do not last as long as they used to and certainly do not take as long to disseminate into the population.

But taken to the logical conclusion, this means that to sustain the city you need a new major innovation every year, month, week, day, hour, minute, second, .... This does not appear to be sustainable. Researcher Geoffry West says: What is the nature of the end stage? We certainly do not have an answer.

Richard Florida, an economic geographer at George Mason University in Fairfax, Virginia, echoes West's concerns. "I worry if we push the speed of the urban organism past that of the human organism," he says. "Already we're hiring personal trainers and coaches to keep us going, and boosting our memory with computers." He adds that as cities continue to get larger, the contrast between the "talent-attracting haves" and the "talent-exporting have-nots" will increase, and the fault lines will not run between the developed and developing world, as they used to, but will split countries internally.
The picture is not very encouraging for the have-nots either. Take the US cities of Buffalo, Pittsburgh and Cleveland, whose populations have been steadily diminishing since 1960. "These are cities that stopped growing because they haven't found the next innovation cycle, and were left with something stagnant or collapsing," Bettencourt says.

Innovations is for all ages

On a lighter note, this weekend's Financial Times has an article called Better Great Than Never which argues that old ages is no barrier to innovation. While acknowledging that innovation is associated with youth in the popular imagination, the article goes to great lengths to demonstrate that it does not have to be so. We copy just one:

William Herschel, for example, was a music teacher before he turned to astronomy, discovering the planet Uranus at 43, infrared radiation and the asteroids at 62, and continuing to make observations into his 70s.

So what can we learn about inventiveness?

So, what are the prerequisites for inventiveness? Youth is not one of them. An exceptional memory is helpful, but ... no guarantee. Scientists of great originality are often poor students, according to psychologist Professor Liam Hudson, writing in The Oxford Companion to the Mind. Fellows of The Royal Society generally have the same class of university degree as less successful researchers. Nobel prizewinners mostly have the IQ of the average undergraduate; and there is no correlation - above a certain minimum, of course - between IQ and achievement ”in any sphere of adult endeavour studied so far”.
Inventors do seem to belong to a certain type, however. They are self-propelling, questioning and persistent. They need, says The Royal Institution’s James, a ruthless objectivity which resists the pressures of received wisdom or religious dogma. A capacity for free association, even fantasy, is important: Einstein prescribed a lively imagination unfettered by reason or logic. In The Act of Creation, Arthur Koestler suggests the key is analogy, bringing unrelated ideas together such as a diver who follows an underwater (i.e. subconscious) chain of which only the two ends are visible above the surface.
Why, then, in spite of the evidence, does it seem strange to speak of old innovators? Because there are impediments to creativity in later life, though not the ones we suppose. They are not biological blocks, but psychological or even cultural ones. People get tired or run out of ideas. Or they fall out of fashion as public taste changes. Or, in a society where youth is prized above everything, they lose confidence in their work, feeling that younger people do not wish to hear what they have to say. ”Perhaps,” says Tallis, ”there is a feeling of moving towards closure, that your life is complete, the job is over, like those individuals who don’t go to the doctor because they feel old and undeserving of attention.”

So whatever your age: innovate today and help your city and human civilization grow. There really are no excuses.

Jump to comments.

Enterprise social software and profits per employee

2007-05-24T12:48:00Z

McKinsey writes about Better Strategy Through Organizational Design and come out strongly in favor of Enterprise Social Software and collaboration techniques:

[T]he new element that can help 21st-century corporations create more wealth is large-scale collaboration, across the entire enterprise, enabled by digital technology. ... Digital technology provides the means not just to promote efficient, effective, and large-scale collaboration but also to measure each person’s “assists” and thus motivate employees to collaborate in ways that were not possible in the past.

Their arguments are closely linked to the profits per employee measure we introduced back in October last year. This is the most important metric for the CEO to manage.

Focusing on this formula (rather than returns on capital and on the amount of capital deployed) offers several advantages. For one, profit per employee, unlike returns on capital, is a good proxy for earnings on intangibles. The reason, in part, is that the total number of employees is easy to obtain, while a company’s capital, surprisingly, is subject to the vagaries of accounting on issues such as goodwill and to corporate-finance decisions such as debt-to-equity ratios, dividend policies, and liquidity preferences. And these days, talent—not capital—is usually a company’s scarcest resource.
Talent is the scarce resource because it is the ultimate generator of the intangibles that drive the creation of wealth in the digital age. Winning companies are those that can increase their profit per employee by mobilizing labor, capital, and mind power into profitable institutional skills, intellectual property, networks, and brands. The returns to companies that can accomplish all this are extremely attractive because intangibles now confer enormous scale and scope advantages. Furthermore, intangibles represent unique assets for the individual companies in possession of them—that is, they are unique in supply—so they can create “natural monopolies,” which are difficult for other companies to replicate.

This is difficult for most executives. Innovation velocity is dependent on collaboration; and collaboration among larger groups and creation networks require different skills and tools than what most executives are used to and is therefore often ignored or placed in the "too hard" category. But it isn't too hard if the executive team is willing to show leadership and embrace new organizational ideas.

Jump to comments.

Need to know versus need to share

2007-05-09T17:50:00Z

If you are into security, classification, and document sharing, then you need to read Jeff Jonas' post "Need to Know" vs. "Need to Share" – A Very Fine Line Indeed. Otherwise you should probably skip it.

A related article on anonymized sematic directories is a great read, not just for people in the security business, but very much for marketing professionals as well.

Jeff, whom I met at the ETech conference, is one of the smartest guys around, and if you are interested in information sharing, data privacy, and mining shared data, then you will be interested in his blog.

Do you have a customer strategy?

2007-05-08T02:03:00Z

I am in Las Vegas for the Unica user conference where we are considering the future of marketing (and how Unica can help you spend a lot of money, naturally). Darcy Bevelacqua from Harte Hanks had a neat line. As a consultant, she gets called in to help companies to define their direct marketing (and customer relationship) approach and to implement it. Her first question is Do you have a customer strategy?, and the answer is almost inevitable yes.

She then goes to Sales, Marketing, and Customer Service and seperately asks the question: Who is your best qustomer?

If she doesn't get the same answer, then she knows you do not have a customer strategy, not matter what your corporate slide-ware says.

She never gets the same answer.

What would the results be in your organization? Why is that?

Jump to comments.

Row over DRM

2007-05-02T12:57:00Z

The BBC is reporting on a row over HD-DVD encryption which was broken almost half a year ago by some customer who actually wanted to watch the movie he (thought he) had purchased.

I think it is terrible that anybody would even think about bypassing digital restrictions management (DRM) technologies, much less actually distribute the 16 bytes that makes up the decryption key in hexadecimal digits or otherwise. Truly terrible. I would never do such a thing. Ever.

The Post-GUI era

2007-04-06T08:28:00Z

The Post-GUI era. I like the expression and I think what it tries to encapsulate is important.

I am back from O’Reilly’s 2007 Emerging Technologies conference. The recurring theme of the conference, at least to my mind, was this: as technology becomes ubiquitous we need to think much harder about how technology interfaces with humans.

You are probably reading this using a computer with a graphical user interface (GUI), i.e. on a screen that uses windows with pictures and text to represent the content and a keyboard and mouse as the input and control elements.

However, when everything has a microprocessor embedded, when everything is a computer, this is not necessarily the best interface. Your microwave oven probably does not have a GUI and it almost certainly shouldn’t have one, despite the fact that it already has significant computing power.

We need new metaphors for user interfaces. The GUI is still for mostly for technically minded people and technical applications, but if computing is to truly become ubiquitous then we need more human, less technical ways of interacting with the technology objects. Casting the technology-savvy “experts” in the role of wizards, Danah Boyd talked about Incantations for Muggles where “muggles” are (apparently) the people without magical powers in the Harry Potter books. How do we make the magic of technology available to the non-magical people?

Magic is one metaphor for how to think about new interfaces, but perhaps a dangerous one. Magic is often hard to understand, hard to learn, and hard to generalize. Maybe ritual religion would be better, but ultimately metaphors are just that and are always going to come with their own problems.

Or how about animism? Humans have always attributed sentience and agency to the external world. The idea that the world is alive, that the objects therein are sentient and can be transacted with, is old and deep and so common to all the cultures of humanity that it may as well be called universal. Until recently, the world was a place of spirits and the places and objects in the external world were identical with these spirits. Can we re-capture and re-purpose this way of modeling the world which seems to be hard-wired into the human brain to a future where all things indeed are if not completely sentient (artificial intelligence still being a hard problem) then at least behaves as if they are and certainly have agency and purpose and the ability to react to our interactions?

Several speakers talked about games as generally useful metaphors. Games are fun, and fun is good, fun works, to mis-quote a popular 1980-ies movie. Games provide feedback on how you are doing. Games provide challenges that are difficult but not impossible. Games allow you to visibly grow in skill and experience. Games are risky. All of this makes for a fun experience for humans. It is a useful exercise to think about what that would mean for your web site.

The world already has an interface, and it isn’t a GUI. We should learn from that.

Two other sessions were particularly interesting: a Birds-of-Feather session with Nilofer Merchant and a couple of presentations from Jeff Jonas on data mining and in particular data mining of anonymized data, but they probably deserve their own entries. Missing out on Kathy Sierra's presentations was a big disappointment.

Enterprise Social Software is Mainstream

2007-03-24T16:53:00Z

Enterprise Social Software has gone mainstream. I say this based on the fact that the analysts are now releasing stacks of research on this area. Forrester is a good example (and one of the better companies out there):

Efficiency Gains And Competitive Pressures Drive Enterprise Web 2.0 Adoption. The article is clear: Web 2.0 technologies are hitting the mainstream both in consumer and business contexts. [...] Almost all of the CIOs that we surveyed recognized Web 2.0 as more than a passing fad. Any questions?

CIOs Want Suites For Web 2.0. The people responsible for technology in the enterprise have strong desire to purchase Web 2.0 technologies — blogs, wikis, podcasts, RSS, social networking, and content tagging — as a suite, as well as an equally strong desire to purchase these technologies from large, incumbent software vendors. This resonates with what I hear elsewhere: I spent Thursday with Oracle in their Redwood Shores headquarters. They traditionally have a good relationship with the CIO/CTO community and their focus is on integration over innovation That is not to say that they do not innovate, but that where a choice has to be made the focus is squarely on solving the integration challenges. (That this suits their aggressive acquisition strategy well is another, not altogether unrelated, matter.)

The message to startup companies in the Enterprise Social Software / Web 2.0 space is very clear: your exit is a trade sale. VCs and entrepreneurs take note.

Social Computing: How Networks Erode Institutional Power, And What to Do About It, is typical of the crop of Web 2.0 “what does it mean” articles. Individuals increasingly take cues from one another rather than from institutional sources like corporations, media outlets, religions, and political bodies, argues the authors, and companies must react by making social computing a strategic asset and by listening more and talking less.

The ROI Of Blogging: The "Why" And "How" Of External Blogging Accountability, suggests that by following a three-step process, marketers can create a concrete picture of the key benefits, costs, and risks that blogging presents and understand how they are likely to impact business goals. It doesn’t get any more mainstream than this.

And in case you think it is only Forrester that are on to this, have a look at How businesses are using Web 2.0 from McKinsey. Yes, that's right, Web 2.0 is now a strategic management issue. They even provide a handy nine-point reference to what constitutes Web 2.0, though it seems designed to show the confusion at McKinsey rather than to really help anybody:

Blogs
Collective intelligence
Mash-ups
Peer-to-peer networking
Podcasts
RSS
Social networking
Web services
Wikis

You get the picture, I’m sure. Web 2.0 is mainstream; old hat. See you at ETech next week to look at the new stuff?

Jump to comments.

Enterprise Social Software Links

2007-02-19T10:08:00Z

We don't normally post just links, but sometimes it is just too hard to keep up with the good stuff. Mike writes from South Africal about lessons learned from social software implementations, and Centopeia (almost as difficult a name as CYBAEA!) comments on Lee's presentation at LIFT.

Love Mike's fourth point:

Like Paul who preached the gospel in the New Testament, it will help you to have a champion - a Timothy or a Stephen - in each town (company) you plant a church (implementation) in. You need a champion. A believer. An evangelist. Someone who will speak for you and walk you through the ins and the outs of the company.

Jump to comments.

Optimal Knowledge Management

2007-02-12T22:17:00Z

Interest in productivity and how to manage innovation and know-how appears to be growing. Our article on employee productivity gets about four times more hits than the next most popular post. I imagine that more and more people in the West are waking up to the fact that innovation and high productivity is the only thing that keeps jobs here (though the link from USA Today probably didn't hurt).

A recent paper from Harvard considers the optimal implementation of knowledge management. With a slight reformatting for clarity:

We derive three main results [about the optimal management of know-how].

First, information about successes is typically more useful than information about failures, since successful methods can be replicated while failures can only be avoided. This supports firms' focus on 'best practice'.

Second, recording mediocre know-how can actually be counter-productive, since such mediocre know-how may inefficiently reduce employees' incentives to experiment. This is a strong-form competency trap.

Third, the firms that gain most from a formal knowledge system are also the ones that should be most selective when encoding information (i.e., the ones that are most at risk from the competency trap); namely, large firms that repeatedly face problems about which there is little general knowledge and that have high turnover among their employees.

Beyond these main principles, we also show that it may be optimal to disseminate know-how on a plant-level but not on a firm-level, and that storing back-up solutions is most valuable at medium levels of environmental change.

I am not sure I am comfortable with the methodology (I prefer practical results and observed measurements to theoretical mathematics, and in any case they could get to the core of the conclusion with much less differential and integral calculus), but their conclusions broadly rings true. Does anybod have any experimental data in this area?

However, on the third point in particular notice the emphasis on formal knowledge systems. I am thinking that a more comprehensive list along their main dimensions would look something like this (click for larger version):

For what we call unknown problems (problems about which there is little general knowledge) the main focus is on collaboration: working together to define solutions to the organization's problems. For companies that are mainly faced with known problems the issue is one of sharing this information effectively.

On the other dimension, with low turnover you benefit from an unstructured approach. The task is mainly a socal one of finding the right people with the right expertise or attitude. An emerging, bottom-up approach is entirely appropriste. But in the high-turnover situation you need a way of structuring th ework to ensure that your relatively inexpereinced workforce are addressing the right problems. You need to impose more of a top-down structure for the collaboration or sharing.

In terms of technologies, sharing can be achieved with document management systems while collaboration is more suitable for a wiki. The social element comes from some way of expressing onself (e.g. blogs) and a way of connecting people (RSS aggregators and lists of experts).

Jump to comments.

Influencers are not influencial

2007-02-07T07:00:00Z

You are not nearly as influential as you think you are. One of my friends are doing research on influencers in mobile networks, and he is going to be crushed (or maybe not) by this entry in a recent HBR list which basically says that influentials are not very ... influential. That is to say that the spread of new ideas in a social network is not dependent on a few super-connected, highly influential members, contrary to popular assumptions. The article offers some specific strategies for marketeers:

Because the ultimate impact of any individual—highly influential or not—depends on decisions made by people one, two, or more steps away from her or him, word-of-mouth marketing strategies shouldn’t focus on finding supposed influentials. Rather, marketing dollars might better be directed toward helping large numbers of ordinary people—possibly with Web-based social networking tools—to reach and influence others just like them.

The results are based on computer modeling, so I still think it will be useful to get some real data from real people. Identifying the influencers is easy (once you get the idea), now we just need the time to analyze their actual influence and the effect of their decisions and recommendations.

Maths is the new India

2007-02-06T20:45:00Z

Maths is the new India. Really! That's what the HBR says in a new list:

The rate at which [mathematical algorithms] are intelligently adopted could be a differentiator in the wider marketplace. Cheap algorithms are like cheap labor and cheap capital—a valuable resource when judiciously employed.

Thought you wanted to know.... And check up on the value of the Perron–Frobenius theorem from a little before AD 1900: $144 billion and counting :-)

Time wasting and internet names

2006-11-26T13:15:00Z

Since it is Sunday I guess we can link to a nice little time-waster that you may already have seen. If you are a Trivial Pursuit fan, read on.

Blufr is a trivia question game. More fun than playing board games with your teenage children ("what's a board game?"), more addictive than cannabis, more informative than the nine o'clock news, playing Blufr is better for your career than suicide and better for your verbal skills than a total laryngectomy.

Go ahead! What have you got to lose?

On a (semi-) serious note, can anybody explain to me the rules for naming internet companies?

CYBAEA Journal

When Big Data Matters

But you can just sample….

When Big Data matters

Business examples

Dealing with Big Data

You may also like these posts:

Commercial Analytics: The Capabilities

Commercial Analytics

Execution

Product, pricing, and proposition

Customer View

Campaign Data

Data Hierarchy

Customer Analytical Record (CAR)

Model Factory

People

Commercial, Customers, and Data

The Analytical Shift

How we can help

Your thoughts

You may also like these posts:

5 common pitfalls of commercial analytics projects

1. Not starting with the business actions

What to do?

2. Not considering all the actions

3. Not experimenting

4. Too much classification, not enough prediction

What to do

5. The ‘always more data’ and ‘data is IT’ traps

What to do

You may also like these posts:

Inflow segmentation – measuring new customers by value not volume

You may also like these posts:

Understanding reasons for churn – and what you can do about it

Learn by doing

You may also like these posts:

Commercial churn modelling

Commercial churn modelling

You may also like these posts:

Why?

You may also like these posts:

Employee productivity revisited

You may also like these posts:

Your mobile phone knows everything about you ... and it is telling

B2B Content Marketing

Marketing lessons from Antiquity

Does social networks influence purchases?

KDD Cup 2009

You may also like these posts:

The financial crisis and physicists

You may also like these posts:

Happy birthday WWW

5 step process for customer base segmentation

5 step process for customer base segmentation

1. Database analysis to determine behavioral clusters

2. Survey your customers to obtain behavioral segment descriptions

3. Determine your market strategy and your company’s position in the market

4. Determine your strategic segments

5. Needs-based segmentation for proposition development

Organizational alignment to make use of segments

You may also like these posts:

3 things we want from a segmentation of the customer base

3 things we want from a segmentation of the customer base

Meaningful segment descriptions

Stable segments for stable customer base

The segments differentiate on the variables that drive our marketing campaigns

You may also like these posts:

The truth about venture capitalists according to Marc Andreessen

Innovation forever

The innovation horizon

Innovations is for all ages

You may also like these posts:

Enterprise social software and profits per employee

You may also like these posts:

Need to know versus need to share

Do you have a customer strategy?

You may also like these posts:

Row over DRM

The Post-GUI era