My 2¢s worth on Big Data

I recently left my role at an Internet market research company. As this company I was helping manage the pre-sales and post-sales for enterprise web-analytics platform business. I worked with unstructured data (collected from web using GET request) for last 9 years and understand the business need and data collection methods in depth. In the past few years “Big Data” has become a buzzword in the field of technology.

To keep this post relevant – I am going to avoid writing things, which you can read on other sources.

Genesis: The idea of big data and projects in this area got popular after Google published a paper on their distributed file system and how it could be used to collect, store and analyze large volumes of data on commodity hardware.

http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf

Purpose: The need to store data in large quantities has been around since banking, telecommunications, airline and power transmission have digitized their data on computers. Typically these cash rich companies would spend on mainframes and ensure high availability costly machines to host this data. These were costly machines and could result in couple of million-dollar worth of hardware, software license and personnel cost. What made it still okay to spend so much was that the data was essential to be stored as each transaction had a commercial value or was recorded for regulatory compliance and thus missing the data was not an option.

http://www.vm.ibm.com/devpages/jelliott/pdfs/zhistory.pdf (Short history of the IBM Mainframes)

Early 2000s: Saw the rise of the internet and online applications where the end user actually was interacting with computers. Thus the purpose of having an computer went beyond record keeping. This also resulted in explosion on the volumes of data generated. While the data in the logs was useful but every single action wasn’t of commercial value; instead there was value in understanding what collection of these logs would tell more about the customer and their behavioral journey.

So this was the challenge that large internet companies like Yahoo!, Google, MSN et. al. were trying to solve. This resulted in creation of systems similar and including the GFS. These systems allowed using commodity hardware for store and querying of data. Thus reducing the cost of maintaining a data collection and analysis system.

My encounter with big data and challenge in learning: As a web analytics consultant I helped companies to collect, ingest and analyze the web traffic logs using software built by companies like Adobe (Omniture/WebSideStory/Visual Sciences), WebTrends, Coremetrics, comScore and Google (analytics). These applications worked nicely to satisfy the reporting needs of the executives and worked as system on the side without interfering the primary ways in which the main core of the web services would work.

Change in recent years: In the recent years internet has become inevitable part of the lifestyle and thus making companies like Facebook, Twitter, Linkedin, Google major part of ones day. This also means that these companies have access to 1bn+ online audiences who can be fed online advertising and thus fueling the online commerce channels. Instead of paying web analytics companies for an analytics system the engineers at these tech companies have resorted to the use of Hadoop(a.k.a. big data) systems to collect, store and analyze the traffic logs.

What it sparked: Since now there is a way to collect, store and analyze hoards of data the application engineers also figured out other ways to store data from clinical research, operations or any other activity which could result in collection of data which was purely logging activity. This data is then mined to perform statistical analysis, predictive analysis, natural language processing, artificial intelligence and machine learning. Such applications provide data analysts with a magnifying glass to look at large volumes or data and find out macro trends and insights, which weren’t possible earlier as there weren’t cheaper ways of performing the analysis and the value of the insights, didn’t generate savings/profits greater than the cost of the systems.

Where its going: Internet of things (IOT) and Mobile technology has facilitated automation of collection of data and thus further fuelling a growth in the collection of more data.

What exactly is big data? There is lot of hoopla about what big data is and what it is not. In simple words it’s a way to store large amount of data, process, query and analyze it using cost efficient hardware system. The software that has become unanimous with big data is Hadoop and other utilities that allow manipulating or querying the data.

How do you explain Hadoop? Hadoop is sort of a misnomer for collection of software and anyone who is knowledgeable about the components will actually be willing to speak specifics about the components. People who are bullshitting their way around will stop at the keyword ‘hadoop’.

•    Hadoop Common: The common utilities that support the other Hadoop modules.

•   Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.

•    Hadoop YARN: A framework for job scheduling and cluster resource management.

•    Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

Once you have installed the HDFS you have a cluster where the file system looks like one big volume/drive but is actually sharded across various units that form your cluster. The tasks written, usually in Java or Python that allow querying the shards and then aggregating the results are called as MapReduce programs. One may say that before the writing the MapReduce logic there is no datamodel to the data – it’s the MapReduce that defines the data model and query model for the underlying data.

Besides this there are couple of other utilities that help you manage the big data system. They can be listed as follows:

·      Ambari: A web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner.

·      Avro: A data serialization system.

·      Cassandra: A scalable multi-master database with no single points of failure.

·      Chukwa: A data collection system for managing large distributed systems.

·      HBase: A scalable, distributed database that supports structured data storage for large tables.

·      Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.

·      Mahout: A Scalable machine learning and data mining library.

·      Pig: A high-level data-flow language and execution framework for parallel computation.

·      Spark: A fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including ETL, machine learning, stream processing, and graph computation.

·      Tez: A generalized data-flow programming framework, built on Hadoop YARN, which provides a powerful and flexible engine to execute an arbitrary DAG of tasks to process data for both batch and interactive use-cases. Tez is being adopted by Hive™, Pig™ and other frameworks in the Hadoop ecosystem, and also by other commercial software (e.g. ETL tools), to replace Hadoop™ MapReduce as the underlying execution engine.

·      ZooKeeper: A high-performance coordination service for distributed applications.

If you are non-technical, data savvy business person then more than likely the “how do you explain Hadoop?” section is where you loose the interest and ignore the details as jibersh. The next thing you will want to do is hire a person who takes care of all the details and run the big data project for you. And now you have a requirement out there in the market looking for 50 business analytics skills with all the technical skills and a person who is in touch with your business for last 10 years. Well if you believe its one person who could do this – then you have it wrong.

In general from my understanding this is how I would divide the big data team:

1)    Make the system work: Traditionally these people have job titles of UNIX system administrators. These people will make the basic infrastructure work and will make the so-called Hadoop file system work with other applications. Their KPI of these resources is ‘system availability’

2)    Business analyst: These people were called business analysts. The key skills for these social people is to find out all the data sources and detail the information contained in these data sources. These people also have to be tech savvy to understand the APIs and data models that allow marrying the datasets for a holistic view of the KPIs on which the organization is run. These resources can usually be the old hands who have been in your company for a while and understand the political boundaries and can negotiate their way to make things happen. People like me who have worked with web data and integrated offline source to create meaningful reporting frameworks can be bucketed here.

3)    Team of analysts: These are the set of people who can write SQL, VBA scripts and excellent skills with creating spreadsheet dashboards and power point presentations.

I have spent some time trying understand the mystery systems and will continue to read more… Like my post is titled, this is my 2 cents worth. Hopefully you enjoyed this post.

Lessons from my motorcycle ride in Ladakh

I just came back from 900KM+ motorcycle ride trip in Ladakh. The route were various hills and passes in the western Himalayas (Ladakh range). The roads are said to be amongst most dangerous roads to drive in the world and Kardung La is the highest motor able pass whereas Chang La is the third highest motor able pass in the world. 

Few things that I learnt through this trip are:
1) Learn to breathe: Breathing happens unconsciously and we don't realise the importance of breathing. At high altitude breathing is a task by itself and has to be performed as a conscious activity and it doesn’t just happen. So before you set to do any other activity you have to remember to breathe! If you breathe (both inhaling and exhaling are important) right then the activity becomes easy. I’m sure this is very important at places with normal altitude as well and it focusing on breathing may allow controlling yourself in emotional, stressful or situation of rage.
On the flip side at 18k feet when breathing was really difficult I also realized what old age must feel like. Breathing becomes difficult task at that altitude and helps understanding the importance of enjoying life while young. Old age is tough and don’t wait to enjoy until you turn old – because the most difficult thing then might be to just breathe ☺

2) Focus on your goals: Motorcycle riding has to start early in the morning to avoid high level of water from the melting snow. Sound sleep is important and the neighbors in tents or hotel rooms may not always agree with your schedule and be noisy. It’s important to focus on your own goals and let others do what they want. They may not always block you from things that you want to do but may be deterring you from focusing on your goals. 
You’ll also find lots of inconsiderate drivers who share road with you and might annoy you and make you lose your temper but if you focus on enjoying your ride and making it to your destination then it becomes a whole lot easier and enjoyable. So if you just focus on your own goals and help colleagues with similar goals to focus then the noise disappears and goals seem feasible.

3) Sometimes journey is equally or more important: Most of the days we went riding there was little to do at the destination. The destination often had awe-inspiring scenery but not activity after dark. I immediately realized that it wasn’t just the destination that was beautiful – the whole journey offered such awe-inspiring views and the motorcycle riding offered freedom to stop anywhere and enjoy and thus making the journey more important than just reaching the destination.

So here are some photos from the trip for you to enjoy and if you agree with above things without going through the same experience then please learn to breathe, stay focused on your goals and enjoy the journey of life!

State of online ad inventory

Three old Jewish friends are sitting around the park, feeding the pigeons and lamenting over how tough things are. "Oy vey," says the one. "Things are so tough in the garment business these days. We have to discount everything by 50 percent. Can you imagine that? Two for the price of one? It's ridiculous!"

"You think that's bad," says the second one. "The watch business is so crazy right now, we literally give away our watches at cost in order to break even."

The third one interjects: "You lucky, lucky bastards. The jewellery business is so bad right now; we give away diamonds for free and throw in $1,000 per transaction in the process!"

The other two look puzzled and say, "How on Earth do you make any money?"

"Don't worry," says the third, "we make it up on volume."

This is an ode to all the publishers out there who have original content but still choose to do business by competing with content aggregators who mooch original content and deploy better content finding strategies and rank better than the ones with the original content. Whats more pitiful - original content owners don't care to provide quality metrics such as the engagement of readers, neither care to measure it nor do much to deploy any content finding strategies.

Decision making without data (gambling)

Decision making without any base data (not 100% of decisions should be taken based on the data - that would turn us into droids) is just like gambling.

George Washington said about gambling and can also apply to decisions made with gut feeling in today's age: "Gambling is child of avarice, the brother of iniquity and the father of mischief. This is a vice which is productive of every possible evil... in a word, few gain by this abominable practice, while thousands are injured."

Avarice-> Only lazy people don't want to put effort in finding data and reason within the data.
Iniquity -> No data - surely does involves politics and biased judgement as there is no rationale.
Mischief -> More a cause of nuisance for the sane and rational people.

A potion made of above three is nothing short of organizational evil and thus can only result in bad things for most and only a select few gain out of it.

Part 4 - Tools for gauging industry competition

For audiences reading this post: please note that I am summarizing a book I am reading and this is not my work. All the content is owned by the author Vaughan Evans. The name of the book is: "Key Strategy Tools". As title of my blog makes it clear these are my cheat sheets so that I can revisit the contents of the book in an easy online format.

---

The five forces ( Porter)
The tool: Competitive intensity determines industry profitability. That is is the basic premise of Michael Porter's work. And he went on to describe in detail what the fundamental forces are which drive competitive intensity.

He set out to show that firms in any industry were constrained from maximizing profit not just by rivalry with their competitors but by four further competitive forces.  These five forces shape competitive intensity:

1. Internal rivalry
2. Threat of new entrants
3. Ease of substitution
4. Customer power
5. Supplier power

How to use it
Internal rivalry: Internal rivalry is shaped by three main sub-forces:

1. The number of players: The more numerous the players, the tougher typically the competition
2. Market demand growth: The slower growing the market, the tougher typically the competition.
3. External pressures: External bodies, in particular government and the trade unions, have great power to influence the nature of competition in many industries. Government regulation, taxation and subsidies can skew both market demand and the competitive landscape. Trade unions can influence competition in a number of ways.

There are other, lesser factors influencing internal rivalry. Barriers to exist are one such such. Season or irregular overcapacity is another factor.

Threat of new entrants: The lower the barriers to entry to a market, the tougher typically the competition. Barriers to entry can be technology, operations, people or cost related, where a new entrants has to:

• develop or acquire a certain technology
• develop or acquire a certain operational process
• gain access to a limited distribution channel
• train or engage scarce personnel
• invest heavily in either capital assets or marketing to become a credible provider

Switching costs also influence entry barriers. The higher the cost to the customer of switching from one supplier to another, the higher are the entry barriers.

Ease of substitution: The easier it is for customer to use a substitute product or service, the tougher typically the competition.

Customer power: The more bargaining power customer possess, the tougher typically the competition. Often this is no more than a reflection of the number of providers in a marketplace, compared with number of customers. The more choice of provider the customer has, the tougher the competition.

Customer power is also influenced by switching costs. If its easy and relatively painless to switch supplier, competition is tougher. If switching costs are high, competition is less tough.

Supplier power: The more bargaining power suppliers possess, the tougher typically the competition. Best of the organizations learn how to duck, dive and survive.

Overall competitive intensity: Mentioned above are the five main forces shaping the degree of competition in marketplace. Put them all together, and you'll have a measure of how competitive your industry is.

When to use it: Always

When to be wary: Some believe that boundary definition - this activity is part of the industry you operate in, that activity is not - can itself place strategy development in straitjacket.

• An indusrty consists of a set of unrelated buys, sellers, substitutes and competitors that interact at arm's length.
• Wealth will accrue to companies that erect barriers against competitors and potential entrants.
• Uncertainty is low, allowing the prediction of competitive response and contingency planning.

---

Assessing customer purchase criteria (CPC)
The tool: Discovering why customers buy is first of three tools on how to assess your firm's competitive position in each of your key product/market segments:

• Identify and weight customer purchasing criteria (CPC) - what customers need from their suppliers in each segment - that is, from you and your competitors.
• Derive and weight key success factors (KSFs) - what you and your competitors need to do to satisfy these customer needs and run a successful business.
• Assess your firm's competitive position - how your firm rates against those key success factors relative to your competitors.

How to use it: Start by asking yourself these questions. What do customers in your business's main segments need fro you and your competitors? Are they looking for the lowest possible price for a given level of product or service? The highest-quality product or service irrespective of price? something in between?

Do customers have the same needs in your other business segments? Do customer groups place greater importance on certain needs?

What exactly do they want in terms of product or services? The higher specifications? fastest delivery? The most reliable? The best technical back-up? The most sympathetic customer service?

Customer needs from their suppliers are called customer purchasing criteria (CPC). CPCs can be usually grouped into six categories. They are customer needs relating to the:

1. Effectiveness of the product or service: The first need of any customer from any product or service is that the job gets done. You the customer have specific requirement on the features, performance and reliability of the product. You want the job done. Not half-done, not over-done, just done. Depending on the nature of the product or service, your criteria may well include: Quality, Design, Features, Specifications, Functionality, Reliability. Some of these criteria will overlap. You should select two to four effectiveness criteria which are most pertinent to customer needs in your industry.
2. Efficiency of the product: The second main customer purchasing criteria heading is efficient. The customer wants to receive the product or get the job done on time. All customer place some level of importance on efficiency for all types of service. Different customer groups may place different levels of importance on efficiency for the same service.
3. Range of products provided: The range of products or services provided is an area customers can find important for some products or services, even most important, and for others of no importance at all.
4. Relationship with the producer: Your supplier does the job and does it quickly. But do you like them? Is that important? The relationship component in providing a service should never be underestimated.
5. Premises - only applicable if customer needs to visit the suppliers premises. Do you need a storefront for your business? What do customers expect of your premises?
6. Price: Set your prices sky high and you won't have many customers. Set them too low and you won't stay in business. Think about the buying decisions you make regularly and the influence of price. For non-essential goods or services, we ten to be price sensitive.

Finding out CPCs: All this is very well in theory, you may ask, but how do you know what customers want? Simple. Ask them! It doesn't take long. You'd be surprised how after just a few discussions with any one customer group a predictable pattern begins to emerge. Some may consider one need 'very important' others just 'important'. But it's unlikely that another will say that it's 'unimportant'.

When to use it: Always

When to be wary: Some customers may have a hidden agenda. They see the meeting as an opportunity get you to nudge down your price. Or to improve your service offering, incurring extra cost, with no increase in pricing. They may rate price as a most important CPC even though they are primarily concerned with product quality.

---

Deriving Key Success Factors (KSFs)
The tool: Key Success Factors(KSFs) are what firms need to get right to satisfy the customer purchasing criteria (CPCs) of the previous tool and run a sound business. Typical KSFs are product or service quality, consistency, availability, range and product development (R&D). On the service side, KSFs can include distribution capability, sales and marketing effectiveness, customer service and post-sale technical support. Other KSFs relate to the cost side of things, such as location of premises, scale of operations, state-of-the-art, cost effective equipment and operational process efficiency.

How to use it: To identify which are the most important KSFs for each of your main business segments, you need to undertake these steps:

• Convert CPCs into KSFs: 1) Differentiation-related 2) Cost-related: We need to work out what your business has to do to meet those CPCs. KSFs are often the flip side to CPC. Functionality may be CPC, so R&D becomes a KSF. Reliability may be a CPC, so quality control becomes a KSF. There's one CPC that needs special attention, and that's price. Customers of most services expect a keep price. Producers need to keep their costs down. Price is a CPC, cost competitiveness a KSF.
• Assess two more KSFs: 1) Management 2) Market Share: There are two more sets to be considered: management and market share. How important is management in genera in your industry? Think on whether a well-managed company, the superb sales and marketing team reinforced by an efficient operations team, but with an average products, would outperform a poorly managed company with a super product in your industry. There's one final KSF - an important one - that we need to take into account that isn't directly derived from a CPC: market share. The larger the relative market share, the stronger should be the provider. A high market share can manifest itself in number of different competitive advents. Once such area is a lower unit cost, but we've already covered this under economies of scale in cost-related KSFs, so we must be careful not to double count.
• Apply weights to the KSFs. You've worked out which are the most important KSFs in your business. Each one has been ranked in order of importance. Now you need to weigh them. A simple quantitative approach works best.
• Identify any must-have KSFs. Are any of the KSFs in your industry must haves? Bear this in mind assessing your competitive position.

When to use it: Always

When to be wary: Don't end up with too many KSFs of you may lose the wood for the trees.
A systematic approach for deriving KSF weightings:
Here's  a step-by-step systematic approach to weighting KSFs:

• Use judgment on the power of the incumbent to derive a weighting for market share of i per cent, typically in the range of 15 to 25 per cent.
• Revisit the importance of price to the customer. If you judged the customer need of medium importance, give cost competitive ness a weighting of 20-25 per cent. If low, 15-20 per cent. If high, 35-plus per cent. If yours is a commodity business, it could be 40-45 per cent, with a correspondingly low weighting for market share. Settle on c per cent.
• Think on the importance of management factors to the success of your business, especially marketing. Settle on m per cent, typically within a 0 to 10 per cent range.
• You've now used up a total of (i+c+m)per cent of your available weighting.
• The balance, namely 100 - (i+c+m) per cent, will be the total weighting for service factors.
• Revisit the list of KSDs relating to service issues, excluding price, which has already been covered. Where you've judged a factor to be of low importance, give it a KSF score of 1. Where high, 5. Rate pro rata for in between.
• Add up the total score for these service-related KSDs (excluding price)=S
• Assign weighting to each service KSF as follows: weighting (per cent) = KSF Score * (1-[i+c+m])/S
• Round each of them up or down to the nearest 5 per cent.
• Adjust further if necessary so that the sum of all KSD weights is 100 per cent.
• Eyeball them for sense, make the final adjustment.
• Check that sum is still 100 percent.

---

Weighing economies of scale

The tool: Size is important. Qualification: Size is important in certain sectors, less so in others.
Another qualification: even in sectors where size is important, small players can survive - if they are nimble.

There are four main economies of scale across the value chain:

1. Purchasing economies - the larger the producer, the more likely they will be able to drive a harder bargain with suppliers.
2. Technical economies - the machinery needed to produce 20,000 widgets a day is unlikely to be 20 times as expensive as that needs to produce 1000 a day.
3. Efficiency economies - the process for producing 20,000 widgets a day is likely to me more highly automated, from handling inputs through manufacturing to handling outputs, and with more advanced or streamlined business processes, for example in R&D, than for the smaller plant.
4. Indivisibility exonomies -  some items are beyond the reach of the smaller producer to buy, whether state-of-the-art equipment.

Economies of scale apply as much to service businesses as in manufacturing and to small as much as to global businesses.

How to use it: How important are economies of scale in your sector? Are there purchasing, technical, efficiency or indivisibility economies?

When to use it: Use it when economies of scale are important in your sector and to your business.

When to be wary: Remember it represents only one of many key success factors. Differentiation may be the name of your game.

---

Corporate environment as a sixth force
The tool: Does government help or hinder your business? Does government, whether local, central or European, through taxation or regulation, greatly influence profitability in your industry.

How to use it: The corporate environment is defined as the combined influence of all external organizations, other than Porter's specified competitors, new entrants, substitutes, customers and suppliers, on the firm.

The corporate environment thus defined includes state and industry-wide bodies such as:

• Central government, through taxation, subsidization, trade restrictions, regulation, employment law, health/safety/environment law, industrial restructuring and even the maintenance of political stability.
• Local government, whether county, borough, region or providence
• National regulatory bodies, such as Ofcom or Ofgem in the UK.
• International regulatory directrives, such as the Basel accords on the capital adequacy of financial institutions.
• Pressure groups, such as industry associations, trade unions, Greenpeace.

The influence of these bodies can greatly influence both market demand and industry competition.

When to use it: When the corporate environment is a determining influence on the profitability in your industry.

When to be wary: When the corporate environment in one of the many factors influencing internal rivalry, such as the number of players or market demand growth, and does not merit recognition as a sixth force.

---

Complements as a sixth force

The tool: Do complements influence profitability in your industry? The other major claimant to the positions of 'sixth force' in the industry competition is complements.

How to use it: Complements are more than just supplies to firm. The value of a assembled PC is intricately bound up with the value of its inter processor - a separate company and one which may extract more value from my purchase than the PC manufacturer itself. This would not be the case with an Apple Mac.

Complements are broader too than direct supplies. An airline depends its profitability not just on the five forces, but on the continuity of operations at its complements - airports, air traffic control, suppliers of aviation fuel, inclusive tour operators.

When to use it: When complements clearly play a major role in driving profitability in a particular industry - for example, agin, in airlines.

When to be wary: When complements play a relatively minor role and do not merit identification as a sixth force.

---

PESTEL analysis
The tool: PESTEL analysis offers a framework for identifying external, often government influenced issues affecting industry competition. It is an acronym of these six groups of issues: political, economic, social, technological, environmental and legal.

How to use it: Examples of the issues covered in PESTEL analysis are:

• Political  - government taxation, legal and regulatory intervention in the marketplace.
• Economic - the macro-economic backdrop, including economic growth, inflation, interest rates and exchange rates.
• Social -  the societal backdrop, including population trends, consumption patterns, age distribution.
•  Technological  - trends in R&D and innovation, affecting both product and production, and the threat from substitute products
• Environmental - trends in weather and climate, and the impact of climate change on your firms operations and customer preference.
• Legal-  trends in laws which impact on a firms operations and decision making, including employment, health/safety/environment, antitrust, customer protection, capital adequacy and governance laws.

PESTEL analysis is sometimes used to throw up the range of opportunities and threats needed for SWOT analysis.

When to use it: PESTEL analysis is an aid to the brainstorming of industry issues. Use it if you are accustomed to it, but be aware of its limitations.

When to be wary: At best the analysis represents a rather unstructured, non-analytical, unranked identification of key industry issues will be as needles in a haystack.

Part 3 - Forecasting market demand

For audiences reading this post: please note that I am summarizing a book I am reading and this is not my work. All the content is owned by the author Vaughan Evans. The name of the book is: "Key Strategy Tools". As title of my blog makes it clear these are my cheat sheets so that I can revisit the contents of the book in an easy online format.

---

Only when you fully appreciate the microeconomic context in which your firm operates and with which it interacts will you have the depth of understanding one which to build a robust strategy. There are two distinct but interrelated aspects of microeconomic behavior to grasp: market demand and industry supply.  The latter deals with the forces driving industry competition and their impart on market share, pricing and profitability in your industry.

The following are a few tools to forecasting:

Sizing and Market-crafting (Evans):
The Tool: Size matters: Without market size you won't know market share. Without market share, you'll find it hard to judge competitive position. The larger your company, the easier it is to find data on market size.  Industry association proliferate and many either compute market share themselves or contract out the job to specialist market research firms.

How to use it: First you must decide what you are looking for: your addressed market or your addressable market. The difference can be huge:

• Adressed market - those to whom you currently offer your goods or services and who may or may not purchase them.
• Addressable (or available) market - all those whom you could serve should you extend your offering.

There are six main ways of sizing a market:

• Top-down market research - start with a known, researched market size and chop out inapplicable sections, or make appropriate assumptions on relevant proportions, to drill down to the target market.
• Bottom-up market research - take disaggregated data from a market research report and assemble the relevant bits that make up your target market.
• Bottom-up customer sizing - estimate how much each major customer spends in the target market and make an allowance for other, minor customers.
• Bottom-up competitor sizing/marketcrafiting - estimate the scale of your competitors in the target market.
• Related market triangulation - use two, three or more known sizes of related markets to gauge a rough estimate on the target market.
• Final triangulation - juggle the estimates from the above sources and subject them to sanity checks; consider giving each estimate a reliability rating, work out relative probabilities and compute a weighted average estimate of target market size.

Market-crafting is one of the approaches mentioned above. There are seven main steps in market crafting:

1. Select your main competitors - those you pitch regularly against, those you exhibit alongside at trade shows - and don't forget the foreign competitors, especially those from lower-cost countries.
2. Take competitor A: do you think you are selling more or less than you into this market? if less, by how much less, very roughly? are they selling half as much you? three-quarters? if they sell more than you , by how much more, very roughly? 10 percent more? A third more? Is there any publicly available information which can guide you on this? - competitor A's sales to this market are unlikely to be available if it is a private company, but employment data can be indicative. What do customers tell you? And suppliers?
3. Taking your current sales level as an index number of 100, assign the appropriate index number to competitor A; if you think they sell less than you in this market, but not that much less, say 10 percent less, give them an index number of 90.
4. Repeat steps 2 and 3 each of the competitors named in step 1.
5. Make an allowance for any other competitors you have not names, those who are small or those who only appear now and gain; this should also be an index number; if you think all these others together sell about half what you sell to the market, give this 'Other' category an index number of 50.
6. Add up all the index numbers, divide the total by 100 and multiply by your level of sales - that is your preliminary estimate of market size. 
7. Ask your staff and contacts to go talk to their friends and contacts who work with the competitors and get their input to refine your estimates.

When to use it Use market crafting or other maker sizing methods whenever you have no their party data source and need to create a market size from scratch.

When to be wary:  Treat the results with caution. Take another look at your numbers. Is there any way they may be right? Do they have access to information you don't? What would that imply for your share, or other competitors' shares?

---

The HOOF approach to demand forecasting
The Tool You have better chance of growing in a market that is prospering than in one thats shrinking. Market size is all very well, but what often matters more in strategy development is what the market is doing, where it is going - the dynamics, as opposed to the statics. Is market demand in your main business segments growing, shrinking, or flat-lining?

How to use it You need to apply these steps for each of your main business segments.
The four steps are:

1. Historic growth - assess how market demand has grown in the past.
2. Drivers past - identify what has been driving that past growth.
3. Divers future - assess whether there will be any change in influence of these and other drivers in the future.
4. Forecast growth - forecast market demand growth, based on the influence of future drivers.

Historic growth - This is where you need to get some facts and figures. If you have access to market research data, whether on a regular basis or with one-off purchase, all your needs should be in there. If not, you may have to do some marketcrafting. Get an average annual (compound) growth rate over a number of recent years, preferably the last three or four. If there have been ups and downs, you should smooth them out with three-year moving averages before calculating the percentage change.

One word of caution: market demand growth is generally measured, analyzed and forecast in real terms.

• In nominal terms: with goods or services priced in the money of the day.
• In real terms: the growth rate in nominal prices deflated by the growth rate in the average prices of goods in that market; as long as correct deflators are used, this growth rate should be a measure of volume growth.

Should you need to business planning and financial forecasting you must bring average price forecast back into mix. Then your revenue forecast, as well as the whole P&L, will be able to be compared directly with market growth-rate forecasts in nominal prices.

Drivers past - Once you have uncovered some information on recent market demand growth, find out what has been influencing that growth. Typical factors that influence demand in many markets are:

• Per capita income growth
• Population groth in general
• Population growth specific to a market
• Some aspect of government policy or purchasing
• Changing awareness, perhaps from high levels of promotion by competing providers
• Business structural shifts
• Price change
• Fashion, even a craze
• Weather - seasonal variations, but maybe even the longer-term effects of climate change

Drivers future - Now you need to assess how each of these drivers is likely to develop over the next few years. Are things going to carry on more or less as before a driver? Or are things going to change significantly. What are the prospects for growth in vertical or complementary sectors? The most important driver is, of course, the economic cycle. If it seems the economy is poised for a nosedive, that could have a serious impact on demand in your business over the next year or two - assuming your business is relatively sensitive to the economic cycle or maybe is relatively inelastic.

Forecast growth - Make sure all drivers are taken into account, irrespective of whether hard data can be found on them. The HOOF process encourages you to seek out all relevant drivers and assess their influence in a structured, combined quantitative and qualitative context.

When to use it: Whenever you need to forecast market demand.

When to be wary: Take care to identify all relevant drivers. Use whatever evidence you can muster dashed by a strong dose of reasoned judgement, in assessing their influence on demand growth, past, present and future. And be wary when computing historic growth rates.

---

Smoothing the moving averages

The Tool:  Where markets have been up and down, showing no consistent trend, take care. Approach to demand forecasting. The best way to deal with the market volatility is to plot a graph on logarithmic paper and draw a line of best fit through the points. Plotting data on logarithmic scale can be challenging for some.

A simple non graphical alternative is to translate the data into moving averages. This enables annual fluctuations to be smoothed out, making it easier to decipher and calculate trend growth rates.

How to use it: Take the set of market data and apply these steps:

• Observe the length of the cycle and select an appropriate time period for smoothing, often a three-year period.
• Take the annual average of values during that time period around any given year.
• Calculate compound growth rates between appropriate start and end points to establish the trend.

Or simply put, the sum of each year's plus the previous year's number plus the following year's number, divided by three.

When to use it: Use it when historic market size data show ups and downs or an irregular pattern.

When to be wary: Don't just compute the numbers blindly. Try to understand what was happening in each of the years to produce such irregular numbers. That will help you avoid the trap of selecting a boom year as the starting point and a bust year as the end-point. Taking boom to boom, bust to bust, average to average period should give you similar answers, but mixing them up can be severely misleading (and is much-loved reuse of the politician! :) )

---

Income elasticity of demand

The Tool: Income elasticity of demand ('IED'). It is a measure of how demand for good (or service) changes in relation to change in the income of customers. It is defined as the percentage change in demand divided by the percentage change in income.

If your business, or one of your product/market segments, addresses a market which is large and generic, you may be able to use IED in your market demand forecasting.

Different types of goods have different IEDs

• Normal good have positive IED
• If an IED is less than 1, the good is a necessity - demand doesn't rise much in good times, nor does it fall back too much in bad times; examples are fresh fruit and vegetables, even tobacco
• If an IED is greater than 1, the good is a superior good - people buy it aplenty in good times, but cut back in bad times; examples are books, meals out.
• If an IED is greater than 2, the good is a luxury - demand fluctuates widely between good and bad times; examples are sports cars, haute couture, holidays in Seychelles.
• If an IED is around 0, the good is inelastic or sticky - demand doesn't change much in relation to changes in income; examples are bread, baked beans
• If an IED is less than 0, the good is an inferior good - demand drops during the good times and returns in bad times: the classic examples are margarine and bus travel.

How to use it: If your market is a whole sector or sub-sector, you may well find an estimate of IED for that sector on the web. To forecast market demand you:

• extract economic forecasts from a reputable source
• multiply by the income elasticity of demand.

When to use it: Use it when your market (or segment) is sufficiently large and homogenous to merit the calculation by some public or academic institution of its income elasticity of demand.

When to be wary: Are you sure your from addresses that whole sector market? After the process of product/market segmentation. The resultant segments tend to emerge as rather specific, both to a product or product type and to customer/end-user group, not to a whole sector market. You may need to revisit your segmentation.

---

Survey methods of demand forecasting

The Tool: There are a number of survey based methods for forecasting demand which you may find pertinent to your strategy development process:

• Survey of customers' intents
• The salesforce estimation method
• The Delphi method
• Pilot test marketing

How to use it

1. Survey of customers' intentions: This is where you chose a representative sample of customers from each major product/market segment and call them up. You ask them what volumes of product they are intending to buy over the next 12,24 months, not from you, but from all competitive suppliers. This can be part of the same survey you will be carrying out to colic it customer views on purchasing criteria and their rating of your firms performance and those of your rivals against those criteria.  But you should not expect spectacular results. Better to accept limited expectations from the outset and tack these questions on to your survey of purchasing people at your customers - which is a necessary, not an optional, component of your strategy development process.
2. Salesforce estimation method: Arrange a debate of participants who are a group of key sales people and debate where they believe market demand is heading. You act as orchestrator and assemble the accumulated opinion of the group. Salespeople get close to their customers and can often be prescient in forecasting a sales stream from their customers. But find it difficult to step back and see the market as a whole - all customers to all suppliers. They spend so much of their lives with each of a handful of trees that wood can become a blur to some.
3. The Delphi method: It is a structured analysis of independent, informed, 'Apollonian' opinion. You reach out to a bunch of reputable, even 'expert' industry observers and ask them a few carefully worded questions on market demand prospects. You collate the replies, but anonymously, and return a summary of the findings to the observers. They look at what others are saying and have the option of sticking to their initial answers or modifying them, with appropriate justification. You amend this summary accordingly and there is your demand forecast, a balanced assessment of the combined wisdom of a handful of Delphic oracles.
4. This is most appropriate where you are introducing a new product or venturing into a new product/market segment. Yours may be a new product or service designed to convey a customer benefit not previously realizable. Write down the key segment needs or product solution that address the market needs in couple of bullet points. Imagine there were many suppliers of your product or service and that the whole country is aware of its existence. What would the market size be? How does that compare with the market size for products or services not a million miles different from what you'll be offering? Does your estimate make sense?

When to use it: Use them when you feel that canvassing of others' views would improve the ragout of demand forecasting.

When to be wary: All forecasting is based on quality of data collection and sources of data. Thus it cannot be relied on completely.

---

Statistical methods of demand forecasting

The Tool: 'There are lies, damned lies and statistics' where the famously damning words of Mark Twain. Perhaps, but basic statistical tools may be of help in your market demand forecasting. The main and simplest ones are:

• Trend Projection
• Regression analysis
• Barometric method (complex to summarize - better look up a good source)

How to use it:
Trend Projection: This involves the placing of a line of best fit through points on a chart. The process is as follows:

• Set out market demand data for each year.
• Plot on logarithmic graph paper, with time on the x-axis and demand on the y-axis.
• Visually place a line of best fit through the points.
• Measure the gradient of the line and that will equate to the average annual rate of growth (percentage per annum).

Continuing the line and seeing where it crosses future years on the x-axis will give you a set of market demand numbers for future years, but they may well be meaningless as forecast. They assume that future demand will be subject to exactly the same influences as in the past.

Regression Analysis: This is a statistical tool which can help you understand how market demand, a dependent variable, will vary in relation to variation in an independent variables, such as GDP or engineering output. The process is as follows:

• Set out market demand data for year.
• Set out the independent variable: for example, GDP, for each year.
• Plot on graph paper, with GSP on the x-axis and demand on the y-axis.
• Visually place a line of best fit through the points (for linear regressions).
• Measure the gradient of the line(m) and the intersect(c) with the y-axis when x=0 and the relationship between demand and GDP will be that of the standard equation, y=mx+c.

Again you need to take great care in using regression analysis for forecasting. The relationship that was evident in the past may not hold firm for the future. An other drivers may play a greater role n the future.

When to use it: Use statistical methods when you suspect that the market crafting approach to demand forecasting would benefit from greater rigor, especially in the computation of pas growth rates.

When to be wary: Take care not to assume, without specified justification, that the trends, regressions or barometers observed in the past will hold true in the future.