Crossover: 2014

Chapter 130 The Pursuing Chaser (Part [-])
Harley Price knew what Eclair Kilkaga called LSTM neural networks.

The more accurate name of the LSTM neural network should be called "long short-term memory neural network".

This is a special kind of recurrent neural network.

Compared with ordinary recurrent neural networks, LSTM neural networks are insensitive to the gap length in application.

This is an advantage of LSTM neural networks that allow them to perform well on longer sequences.

Harley Press carefully recalled the algorithm features described in the technical route of LIN HUI and the actual performance of the software Nanfeng APP.

The technical route mentioned in the generative summary algorithm of LIN HUI clearly stated that the text information should be serialized and marked through vectors before further processing.

The LSTM neural network happens to be able to handle long sequences of text.

The algorithms tinkered with by LIN HUI are extremely accurate when dealing with text summarization.

The long-short-term memory neural network has a high accuracy when dealing with practical problems.

LIN HUI relies on the generative summarization algorithm and Nanfeng APP can only process one news summary at a time.

The shortcoming of the long short-term memory neural network is that the structure is relatively complex, and there are disadvantages in parallel processing.

If the algorithm of LIN HUI is only one aspect, it is more consistent with the characteristics of the long-term short-term memory neural network.

Harley Price might think it was just a coincidence.

But now, three small probability events come together.

Harley Price felt it was no mere coincidence.

He felt more and more that Eclair Kilkaga's deduction was correct, and he couldn't help but sigh:
"I asked why the neural network used in the LIN HUI algorithm has the shadow of a cyclic neural network but is somewhat different from the traditional cyclic neural network. It turned out that the long-short-term memory neural network was used.

Eclair Kilkaga really has you!Can think of this direction.

To be honest, I thought the neural network characteristics applied in the LIN HUI algorithm were a bit weird at first.

But I really haven't thought about the direction of the long-term short-term memory neural network..."

Eclair Kilkaga can understand why Harley Price didn't think of LSTM in the first place.

In recent years, long-short-term memory neural networks are mainly used for speech recognition.

At this stage, ordinary researchers really don't think of using long-term and short-term memory neural networks in text summarization.

However, in theory, it is entirely feasible to use long short-term memory neural network neural network for text recognition.

But how to apply the long short-term memory neural network to text recognition is not clear to Eclair Kilkaga for the time being.

This will take some time to explore.

Harley Price didn't think there should be another reason for LSTM neural networks.

Because LSTM neural networks are nothing new.

Hochreiter and Schmidhuber proposed long short-term memory neural network in 1997.

It has been nearly 20 years since today, which can be said to be a long time ago.

Although the long short-term memory neural network may have certain advantages when dealing with long sequences of text.

But in fact, the long short-term memory neural network was not proposed for text processing.

The reason why the long-short-term memory neural network was proposed at that time was to deal with the gradient disappearance and gradient explosion problems that may be encountered when training traditional recurrent neural networks.

In machine learning, when training artificial neural networks with gradient-based learning methods and backpropagation.

Sometimes the problem of vanishing gradients and exploding gradients is encountered.

Neither scenario is what the researchers hoped to see.

After the gradient disappears or the gradient explodes, the original deep learning cannot be deep at all, it can only be said to be shallow learning.

In some extreme cases, not to mention shallow learning, even the most basic machine learning cannot be done.

All in all, the vanishing and exploding gradient problems can greatly reduce the training efficiency of deep learning with neural networks.

The problem of gradient disappearance and gradient explosion is also an extremely difficult problem.

Researchers related to neural networks noticed the phenomenon of gradient disappearance and gradient explosion in 1991.

This problem has been alleviated to some extent after the emergence of long short-term memory neural network.

However, the problem of gradient disappearance and gradient explosion has not been completely solved.

In addition to using long-term short-term memory neural networks, there are several other ways to deal with the problem of gradient disappearance and gradient explosion (such as multi-level hierarchy, using faster hardware, using other activation functions, etc.). Each limitation.

In short, the problem of gradient disappearance and gradient explosion has not been completely solved.

Today, the problem of gradient disappearance and gradient explosion has become a dark cloud in the sky of machine learning.

This problem has seriously restricted the development of machine learning.

Thinking of this, Eclair Kilkaga couldn't help feeling a little emotional.

I don't know when this problem will be completely solved by who? ? ?

Eclair Kilkaga suddenly felt that there was no need for him to be so serious about the LIN HUI algorithm?

Problems such as gradient disappearance and gradient explosion have not been completely resolved for more than 20 years.

No one is in a hurry?At least it seems that no one is in a hurry?
Why do I have to compete with such an algorithm as LIN HUI?

Eclair Kilkaga felt suddenly tired.

But in the face of excited colleagues, Eclair Kilkaga is not going to retreat.

Eclair Kilkaga: "I'm still not sure that the LIN HUI algorithm is using a long short-term memory neural network.

It can only be said that the characteristics of the neural network used in the LIN HUI algorithm are somewhat similar to the long short-term memory neural network.

Whether it is true or not has yet to be verified.

It's a loss to say that those high-level executives broke up with the MIT Natural Language Processing Text Summarization Research Group.

As far as I know, Eve Carly and the others used recurrent neural networks when they were researching extractive text summarization algorithms.

It's just that the specific recurrent neural network they used is not yet clear.

But anyway, I think it would be a big help for us to have the help from MIT. "

Harley Price: "That's a problem, but not a big one.

The most indispensable thing in country m is research institutions.

Some time ago, I heard that Nick guy said that Professor Jules of Princeton University was working on a recurrent neural network project.

Maybe we could start a collaboration with Princeton University? "

Eclair Kilkaga: "Um, are you sure you want to deal with those arrogant math dudes in Princeton?
They see us the same way we see those liberal arts students?
And if we cooperate with them, who will lead?How are the results of the research divided? "

Harley Price: "It doesn't matter what they think of us.

A group of people engaged in mathematics are now engaged in recurrent neural networks. Who is more dominant?

As for who takes the lead, let’s talk about it later, all those who have achieved it will be respected. "

Eclair Kilkaga: "Then you go to contact, anyway, I am too lazy to negotiate with that old bald Jules."

Harley Price: "Well, I don't really want to contact Jules either..."

Eclair Kilkaga: "Then you still have this bad idea?"

Harley Price said nastyly: "Maybe we can ask Asile Velasquez to go, who told him to sell the patent to that LIN HUI..."

Eclair Kilkaga: "That's a great idea!"

　There are several ways to deal with gradient disappearance and gradient explosion, the residual neural network mastered by the protagonist.The residual neural network can not only deal with this problem, but also solve a very important problem by the way.The paper that proposed the residual neural network has a particularly large influence.It has been less than six or seven years since the paper was put forward, and now it has been cited 107599 times.It feels like the protagonist took out this thesis, and the doctor is fine.And the paper on the normal timeline appeared in 15/16.Not too outlandish.

　　
　
(End of this chapter)