Redefining the production process through search intelligence

Most people working on the internet with a bit of commercial awareness understand the need to make their products or content findable both by search engines and users. With the majority of website visitors often coming direct from search engines, if people can’t find your business (commercial or otherwise), it is unlikely to survive too long without significant marketing and/or word of mouth publicity.

In the BBC's case, you could even argue that one of the reasons its web operations needed to be rationalised was because so many ‘sites’ were produced without any consideration for how findable they might be in search, resulting in some lacklustre weekly stats.

Once upon a time

Historically, a BBC proposition might have come about through various routes: a connection with a TV programme (now done much more intelligently through /programmes); a season; a learning drive; a strategic partnership; a commissioning need; or simply because it’s what we’ve always done. Along the way, audience research might have been commissioned and personas written, marketplace analysis performed and prototypes tested etc. But in all of this, nowhere, not least at the sharp end of content production or audience planning would anyone have spent too much time studying search demand.


Keyword research, evolved

Recently, I’ve been working with a small team of specialists to help communicate, train and evangelise to BBC production teams about best practice SEO among other aspects of web development. One aspect of this has focused on the keyword research process. One of the main aims of this is to increase the product value (or the amount of licence fee spent), ensuring that it reaches its intended audience.

Where once a Producer might have justified their lack of SEO awareness by saying “Well I Googled BBC [insert site name] and it came top so I figured we’re all sorted” we like to think we’ve progressed from there now. In addition to these mostly 'run of the mill' optimisation tactics, we’ve been contributing our search data insights to a forthcoming Knowledge and Learning product.

While recommending that Producers use the Google Keyword Tool and Google Insights for Search, before they even build anything, there’s some much deeper analysis we’ve been doing, which shows considerable promise for helping inform future content strategies and ways of working.


How do search demand insights help us build and maintain better products?

At the simplest level an SEO practioner might say keyword research allows your pages to perform better in search by using precisely targeted words. To me that’s slightly different to the type of search demand analysis our team (Duncan Bloor mostly) have been doing of late.

So what are the benefits of search demand analysis?:

• Helps you form an understanding of your audience needs. What are people looking for? What language do they use? Am I meeting those needs? If not, what can I change to do so? 
• Helps inform the structural design of a product, whether that’s modelling a domain, devising a url strategy, or labelling a navigation system
• Helps you see relative popularity of one content topic over another, which in turn helps you tailor the content offer to fit demand, (eg ‘Why are we offering all this content about the Middle Ages when the appetite for Victorian History is ten times greater?’)
• Allows you to compare data against insights gained from other marketing and audience research activities or more traditional web analytics data 
• It offers a broad and immediate snapshot of a subject -  (NB - useful when a company is going through significant change, with people leaving, moving or with limited knowledge about a new content area)
• You can spot ongoing demand compared to seasonal trends, allowing you to better promote your content at the right time, in the right places 
• It’s real. While the output of the research is certainly more Art than Science, you simply cannot argue with the data 
• It’s flexible. No desire to create content about Victorians? What about Royal Weddings, or Code Breaking?  

So why isn’t everyone doing this?

Put simply, extracting the gold from them there spreadsheets is time-consuming. It’s graft, and it’s easy for thousands of rows of data to blend into one big meaningless blob. It takes a combination of an analytical mind; editorial judgement; marketplace awareness; and above all the ability to tell a story and then shape it into something that can be easily interpreted at a glance, often by skeptical minds, quick to pick up on the smallest anomaly or quirk in the data.

Seen in isolation, it’s not going to get you far either. Instead we see it as a strong foundation on which to build a happy house, sensibly designed with inviting rooms, each telling a unique and desirable story in a way that captures the imagination, and that people want to tell all their friends about, and want to return to.

The method 

Say we’re interested in building a History product (you might not be - but the process is of course the same). Outlined below are the steps we go through to gain search demand insights. You'll need Experian Hitwise data, combined with brief checks using Google Insights and any good keyword discovery tool. Broadly, the steps are:

1. Identify other websites relevant to you in the History market. If Hitwise doesn’t already have a category of sites you need to define a category by identifying who is already performing well for the most obvious terms. The more sites the better, but they must be relevant to your interests. This can take time and will require constant checking as some sites might skew your data unfavourably (eg sites covering a broad set of topics are less useful than more focused ones.
2. Download a large set of data from a reasonably long time period.
3. Once the data has been cleaned, run a card sort on the top 100 or so terms. This might result in historical categories such as Kings and Queens, Famous People, Wars, Victorians etc. This is also time-consuming  as you research an area in great detail. For example, do searches around “Victorians” belong in “a category called “Victorians” because the search demand there is so great, or shall we lump them in with a broader period in time. If so, how broad should that period be? (At this stage it's helpful to get help from someone with strong knowledge of the subject matter)
4. Total up your groups in terms of search volume and compare their relative sizes (as in the chart below) 
5. Visualise it using your software package and method of choice (you could use wordles combined with something like the graph below)

And, in a bit more detail (large version on Flickr):

It’s important to understand that the output is a resource that can be revisited and sliced ‘n diced in many different ways. This is really just the starting-point for discussion and hopefully an inspiring first-cut.

 UK Search demand for  History

Here we can see that Romans accounts for 9% of the UK history search market (according to our defined Hitwise category). 

Wordle showing top terms for searches around Romans

When we sift out some of the more obvious words (Kids, History, Ancient etc) we can gain insights around where the biggest demand lies. Food, recipes, clothes, goddesses, soldiers might be avenues worth exploring. And no doubt there are some more niche areas too that could be transformed into unique content.

We can then compare that to what we get from Google Insights.

Of course any search insights and recommendations then need to be weighed-up against numerous other questions:

• Does it fit with the overall content strategy?
• Does this capitalise on a user need to learn more about something featured in a programme?
• Does this allow us to bring the BBC archive to new audiences?
• Does this meet a specific learning outcome?
• Can we bring anything new, and of high quality to this topic that is not already well-served elsewhere?
• What legacy content do we already have that performs well, and if there is any, can we afford to do anything to improve it? 

How can we translate this demand into a compelling story? This isn't about using keyword research to stuff the right words into pages, but instead as a springboard for discussion around content commissioning or product design.

If we compare this activity to things like focus groups or user research, to my mind, this adds easily as much value, not just because the pool of data us so vast (millions of internet users), but because it’s telling us what people are actually looking for, not what they told us they might like.

Where do we go from here
The next step is to figure out how we weave these kind of insights into the production process. I may post something on that at a later date.

For more inspiration about the value of tapping into search and analytics, check the links below.

• More about this work: Our thirst for knowledge, from Duncan Bloor
• Guardian: The metrics are the message: how analytics is shaping social games
• Bank of England: Using internet search data as economic indicator
• BBC News: House price changes 'predicted' by web search data
• Google: Cartoon about Flu activity vs search behaviour