Emerging Tech

Google Is Working On A New Type Of Algorithm Called “Thought Vectors”

Google a step closer to developing machines with human-like intelligence

Professor Geoff Hinton, who was hired by Google two years ago to develop intelligent operating systems, said that the company is on the brink of developing algorithms with the capacity for logic, natural conversation and even flirtation.

The researcher told the Guardian that Google is working on a new type of algorithm designed to encode thoughts as sequences of numbers – something he described as “thought vectors”.

Although the work is at an early stage, he said there is a plausible path from the current software to a more sophisticated version that would have something approaching human-like capacity for reasoning and logic. “Basically, they’ll have common sense.”

The idea that thoughts can be captured and distilled down to cold sequences of digits is controversial, Hinton said. “There’ll be a lot of people who argue against it, who say you can’t capture a thought like that, he added. But there’s no reason why not. I think you can capture a thought by a vector.”

Hinton believes that the “thought vector” approach will help crack two of the central challenges in artificial intelligence: mastering natural, conversational language and the ability to make leaps of logic.

He painted a picture of the near-future in which people will chat with their computers, not only to extract information, but for fun – reminiscent of the film, Her, in which Joaquin Phoenix falls in love with his intelligent operating system.

“It’s not that far-fetched,” Hinton said. “I don’t see why it shouldn’t be like a friend. I don’t see why you shouldn’t grow quite attached to them.”

In the past two years, scientists have already made significant progress in overcoming this challenge.

Richard Socher, an artificial intelligence scientist at Stanford University, recently developed a program called NaSent that he taught to recognise human sentiment by training it on 12,000 sentences taken from the film review website Rotten Tomatoes.

Part of the initial motivation for developing “thought vectors” was to improve translation software, such as Google Translate, which currently uses dictionaries to translate individual words and searches through previously translated documents to find typical translations for phrases. Although these methods often provide the rough meaning, they are also prone to delivering nonsense and dubious grammar.

Thought vectors, Hinton explained, work at a higher level by extracting something closer to actual meaning.

Ascribing Each Word A Set Of Vectors

The technique works by ascribing each word a set of numbers (or vector) that define its position in a theoretical “meaning space” or cloud. A sentence can be looked at as a path between these words, which can in turn be distilled down to its own set of numbers, or thought vector.

The “thought” serves as the bridge between the two languages because it can be transferred into the French version of the meaning space and decoded back into a new path between words.

The key is working out which numbers to assign each word in a language – this is where deep learning comes in. Initially the positions of words within each cloud are ordered at random and the translation algorithm begins training on a dataset of translated sentences.

At first the translations it produces are nonsense, but a feedback loop provides an error signal that allows the position of each word to be refined until eventually the positions of words in the cloud captures the way humans use them – effectively a map of their meanings.

Hinton said that the idea that language can be deconstructed with almost mathematical precision is surprising, but true.

“If you take the vector for Paris and subtract the vector for France and add Italy, you get Rome,” he said. “It’s quite remarkable.”

Dr Hermann Hauser, a Cambridge computer scientist and entrepreneur, said that Hinton and others could be on the way to solving what programmers call the “genie problem”.

“With machines at the moment, you get exactly what you wished for,” Hauser said. “The problem is we’re not very good at wishing for the right thing. When you look at humans, the recognition of individual words isn’t particularly impressive, the important bit is figuring out what the guy wants.”

“Hinton is our number one guru in the world on this at the moment,” he added.

Some aspects of communication are likely to prove more challenging, Hinton predicted. “Irony is going to be hard to get,” he said. “You have to be master of the literal first. But then, Americans don’t get irony either. Computers are going to reach the level of Americans before Brits.”

A flirtatious program would “probably be quite simple” to create, however. “It probably wouldn’t be subtly flirtatious to begin with, but it would be capable of saying borderline politically incorrect phrases,” he said.

Many of the recent advances in AI have sprung from the field of deep learning, which Hinton has been working on since the 1980s. At its core is the idea that computer programs learn how to carry out tasks by training on huge datasets, rather than being taught a set of inflexible rules.

With the advent of huge datasets and powerful processors, the approach pioneered by Hinton decades ago has come into the ascendency and underpins the work of Google’s artificial intelligence arm, DeepMind, and similar programs of research at Facebook and Microsoft.

Hinton played down concerns about the dangers of AI raised by those such as the American entrepreneur Elon Musk, who has described the technologies under development as humanity’s greatest existential threat. “The risk of something seriously dangerous happening is in the five year timeframe. Ten years at most,” Musk warned last year.

“I’m more scared about the things that have already happened,” said Hinton in response. “The NSA is already bugging everything that everybody does. Each time there’s a new revelation from Snowden, you realise the extent of it.”

“I am scared that if you make the technology work better, you help the NSA misuse it more,” he added. “I’d be more worried about that than about autonomous killer robots.

Subscribe To WT VOX Newsletter

Subscribe To WT VOX Newsletter

Join our mailing list to receive the latest news from wearable tech, fashion tech and all emerging technologies.

Thank you for subscribing to our newsletter.

6 Comments

6 Comments

  1. Phil

    22nd August 2015 at

    My first thought on reading this was of Kurt Godel… if this approach is successful., will it finally prove that language is either incomplete or inconsistent? 🙂

    • Rob Freeman

      24th August 2015 at

      Hey, great comment. I started a Google Group to discuss just this idea back around 2007. The group is called Grammatical Incompleteness. No posts for a long time, but feel free to try and wake it up.

      The idea was that yes, language is incomplete or inconsistent when viewed as a formal system of rules or a grammar, and this explains why we have failed in our attempts to get computers to understand normal language up to now.

      There’s a corollary that it should be easy to do so if we change our expectations, but we probably need to go beyond the learning techniques Hinton is using to do it.

      • Mayank

        28th August 2015 at

        Sanskrit is considered to be a grammatically complete language. Too bad no one uses it.

  2. Mr E

    23rd August 2015 at

    this is amazing

  3. amirouche

    30th August 2015 at

    This seems to be similar work as word2vec explains on «Learning the meaning behind words » http://google-opensource.blogspot.fr/2013/08/learning-meaning-behind-words.html

    • test

      14th March 2016 at

      exactly … seems like word-embeeding

Leave a Reply

Your email address will not be published. Required fields are marked *

To Top
SUBSCRIBE TO WT VOX NEWSLETTER

SUBSCRIBE TO WT VOX NEWSLETTER

Join our mailing list to receive the latest news from all emerging technologies.

 

Thank you for subscribing to our newsletter.