The online video service that's getting personal

In Todd Yellin's ideal world, technology can read your mind.It may seem intrusive, but in an age when Google can gather vast quantities of information about your interests for online advertisers, and Facebook may know more about your family's activities than you do, Yellin just wants to know which film you would most like to watch.

A former film-maker and critic, he works for the US on-demand video service Netflix, developing the firm's "personalisation" technology. It determines which films and television shows users are presented with when they log in.

"We'll be finished when I can read your mind and we show one title up there and it's exactly right every time," he says. "That's utopia."

Such perfect personalisation is a white whale of Silicon Valley. Major web firms such as Amazon, Google and Facebook, have legions of engineers and mathematicians trying to work out what users want without their systems having to be explicitly told. They analyse online behaviour by designing software algorithms - each one effectively a machine designed to draw conclusions from complex data.

At Netflix, which arrived in Britain in January, personalisation is central, partly because it relies on showing users more obscure or older films and televisions shows. They are cheaper to license than the blockbusters for which the likes of BSkyB pay big prices. At pounds 5.99 per month for an unlimited service available via the web, connected TVs, games consoles and mobile devices, Netflix can't afford to do that.

"We are going to pick the best ones we think each person might want to watch and show them to them," says Yellin. "Users' tolerance for scrolling through lists of content is only a few dozen titles, so we want to get them right away."

Netflix begins its effort to read your mind from the moment users sign up, with a round of 20 questions. New members are asked what genres they like - action, romance, whatever - to build up an idea of their tastes. However, what people say they like and what they actually watch are not always the same, Netflix has discovered, and it is the latter that matters more in the long run.

"I could walk around at a cocktail party and say 'Oh yeah, I like all these foreign documentaries,' and then, in the privacy of my own home, I'll hit play on Paul Blart: Mall Cop," says Yellin.

"So in a matter of weeks of you joining, we start using implicit information more and more. We throw information about what you've watched, how long you watched it for, how quickly you watched the next episode of something and a whole bunch of other stuff into our algorithms to produce your personalised recommendations, which will change over time."

Dozens of algorithms are used to produce the rows of recommendations that Netflix shows members, such as a personal top 10, content similar to films they have recently watched, and personalised genres. The latter can be absurdly specific, such as "feel-good, opposites-attract comedies".

"Personalisation is hard because it involves big data and it needs to respond incredibly fast," explains John Ciancutti, who leads the engineers who code the algorithms via a never-ending cycle of incremental improvement, with trial-and-error testing on unwitting groups of members. "We need to calculate new personalisation for every single user all the time, and that's an immense engineering challenge."

Ciancutti, himself an engineer, is perhaps less of a dreamer than Yellin, although both agree that perfect personalisation is nowhere near a reality.

"It'll never be the case where we know exactly what you want to watch. Each of us has different tastes and interests at different times; you might have a boyfriend or girlfriend over, or whatever," says Ciancutti.

"After 100 years, people are still coming up with incredible innovations in car engines. We're maybe five or 10 years at most into online video in any meaningful way. People will still be working on personalisation 100 years from now, because it really is our engine."

Silicon Valley's obsession with personalisation technology has attracted critics, who argue that it cuts users off from new types of information and serendipity. The term "filter bubble" has been coined to describe this isolating effect.

Netflix says its new Facebook integration will help combat this effect by connecting users' accounts, and so allowing them to see what their friends have been watching. It is early days for the system, but Netflix is bullish on the impact that humans, as well as algorithms, can have on viewing rates.

"Ultimately the best metric to measure customer satisfaction is to measure whether people are sticking with us month after month," says Yellin.

"In that sense we couldn't care less about what users watch."