Big data in China

Chapter 21 Big Data and Thinking Change (2)
In the era of big data, some cumbersome data management processes may only be solved by a "cloud".For example, a customer order usually needs to go through multiple enterprise data management systems such as ERP, supply chain management, product data, and inventory.This is very complicated, and its process is like human thinking. It needs different processes to realize it, and finally completes the choice and takes action.

At present, the data management model of most enterprises in China still adopts independent and decentralized systems.As someone who studies big data said: "If there is a suitable cloud storage system, these data can be integrated together, so that the operation of the enterprise can be seen at a glance."

Cloud storage is the prerequisite for realizing these functions. It simplifies complex issues, and it is also the concentrated expression of big data thinking, which simplifies all complex issues.When we understand big data, we must absorb this precious nutrition, let it penetrate into our new thinking, and solve problems with centralized and simplified thinking.

Let's consider Buffett's advice: simple is better than complex.Buffett can often explain cumbersome investment issues with the simplest logic, such as his value investing philosophy.Why do most of us not like simplicity and complexity?The simpler the truth, the more complicated the fallacy.The ways of success in this world show that any real philosophy of success is usually not very complicated, but a very simple system.Not only in terms of investment philosophy, but also in the philosophy of life, and more importantly, in the management and production control of enterprises.For example, cloud storage, which can bring more convenient and accurate results.

In terms of efficient and standardized management of office documents, cloud storage is also promising.First of all, it is necessary to build a server, and then a special person will carry out daily maintenance, but the investment is high, the operation is complicated, and the professionalism is strong.Even if multiple modified versions of an industrial design are distinguished by different file names, they may still be confused or even lost. Especially for a specific industry such as industrial design, the standardized management of office documents may be more important.If an enterprise wants to build its own data center to integrate data, it only needs to rent a space on the central server according to its own needs to realize cloud storage and computing services.

Buffett said that his approach to value investing is very successful, but very simple.How easy is it?It is so simple that none of the "three highs" are required: first, advanced mathematics is not required, second, high education is not required, and third, high IQ is not required.These three complicated things are useless because they deviate from the truth of investment.

He said: "I have never found that advanced mathematics has any role in investing. As long as you know some elementary school arithmetic is enough. If advanced mathematics is required, I have to go back to deliver newspapers. I have never found that in investing. What is the role of advanced mathematics. To invest successfully, you don't need to know any professional investment theory. In fact, it is better for everyone to know nothing about these things. Investing is not a game of intelligence. People with high IQs may not beat those with low IQs people."

There is a sentence in Laozi’s "Tao Te Ching", which is called "Tao can be said, very Tao." He also said: "My words are very easy to know and very easy to do. No one in the world can know, and no one can do it." This is what Buffett said.A person in charge of an industrial design company said to me: "Using the traditional storage mode that relies on file names to distinguish is easy to cause version confusion. Coupled with factors such as the flow of designers and collaborative design in different places, it often leads to design results. lost."

Therefore, industrial design is an industry that needs to deal with text data for a long time.The finalization of a design achievement needs to be revised repeatedly by designers. After going through multiple links of design, modification, review, discussion, re-modification, and re-review, many design versions are produced.So is it still possible to make a profit?It is very difficult. Under the leadership of this design thinking, profits will be squeezed to a minimum.

For any industry or business, if you want to get the maximum profit, there are only two words: simple.

Can be imprecise, must be as many as possible

When we extract or process data from a technical level, thinking confusion will also occur.In fact, the origin and type of confusion are originally "a mess."For example, when we use Twitter information for sentiment analysis to predict Hollywood box office, there will be some confusion.

Among them, the manifestation of confusion is actually the inconsistency of the format.If we want to achieve a consistent format, we need to carefully clean the data before data processing, which is difficult to do in the context of big data.

In order to scale, we tend to accept the existence of a moderate amount of errors, of course, including errors of thinking.As Mr. Kayer, a technical consultant, said to me, sometimes 2 plus 2 equals 3.9, which is not bad.It is worth noting that error is not inherent in big data itself.It is simply a flaw in the tools we use to measure, record and communicate data.

Big data does not need to be sampled to obtain the final result and the final law.Because the data it obtains is the overall sample data, which is analyzed and summarized from huge sample data, it can allow inaccuracy, but it must have a sufficient amount of data.Moreover, it does not require the source of the data (such as the user) to answer any specific questions, but actually obtains the "all behaviors" of the user, records all their information, and copies them all exactly to become a Analyzed reference data.

Big data not only makes us no longer expect precision, but also makes it impossible for us to achieve it.Of course, the data cannot be completely wrong, but we are willing to make some compromises on accuracy in order to understand the general trend.If the technology ever becomes perfect, the problem of imprecision will cease to exist.Errors are not an inherent feature of big data, but a real problem that needs to be dealt with urgently, and may exist for a long time.Today, the benefits brought to us by big data allow us to accept the existence of inaccuracy.

Suppose you want to measure the temperature of a vineyard, but there is only one temperature gauge for the entire vineyard, then you must ensure that this gauge is accurate and works all the time.If it becomes ten or even hundreds of measurements per minute, not only the readings may be wrong, but even the time may be confused.So we sacrifice precision for a wider set of data, and see details that otherwise wouldn't have been noticed.If we measure the temperature every 1 minute, we can at least ensure that the measurement results are ordered in time.

Just imagine, if information flows in the network, then a record is likely to be delayed during the transmission process, or even completely lost in the rushing information torrent, and it is meaningless when it arrives.While the information we get is no longer accurate, the sheer volume of information collected makes it more cost-effective for us to forego strictly precise choices.

Also suppose that if there is one meter for every 100 vines, some of the test data may be wrong, but the many readings combined can provide a more accurate result.The value it provides can not only offset the impact of wrong data, but also provide more additional value.Because it contains more data, it will not be more confusing.

We traded precision for high frequency and observed changes that might otherwise have been missed, Kayer said.While these mistakes can be avoided if we put enough effort into them, in many cases we can do more good by being tolerant of them than by working to avoid them.

Sometimes, when we have a lot of new types of data, precision is not so important, and we can also grasp the trend of things.This is another shift in focus, as before, statisticians have always been interested in improving the randomness of the sample rather than the size.Because when we transform data, we are turning it into something else.

However, in addition to initially contradicting our intuitions, we are able to make better predictions and understand the world better by embracing the imprecision and imperfection of the data.Because the business benefit of having more data far outweighs a little bit of accuracy, we usually don't go to great lengths to improve the accuracy of the data.

The non-standard nature of big data forces us to pay attention to efficiency but not to pursue extreme precision.

●Be aware that 95% of data is non-standardized, and 5% of data is standard structured data.

●Big data processing must consider all data and must accept non-standard data, and parts cannot replace the whole. An inevitable process of data analysis is to standardize the mixed non-standardized data.

• Labeling on the web is an example of good imputation to normalized data.So people need to collect complex data.

☆Descriptive analysis
What is descriptive analysis?In layman's terms, it is the reports, icons, statistical charts, etc. that we often see.We look to descriptive analytics to understand what happened in the past, why it happened, what is happening now, and what will happen in the future.Then think rationally, what kind of things do I want to do, what do I want to happen in the future, and I can make this happen in the future.

That is to say, in the best case, we can use descriptive analysis to make some kind of predictions about the future, and the accuracy of the predictions can be guaranteed.

☆Real-time

For any data, real-time performance is very important.It is not just a large category of thinking and methodology, but real-time performance must be more important than absolute accuracy.The well-known shopping basket analysis is to make a relatively accurate analysis based on historical data.The best time is when the user is still browsing and looking for things, not when they are finally checking out, so this is a very practical question that you can think of when you are shopping in the supermarket.

(End of this chapter)