Dialogue with Zhu Jun, Institute of Artificial Intelligence, Tsinghua University: What security risks are hidden in the popular AI model?

Text: Li Haidan, Tencent Technology

Image source: Generated by Unbounded AI

Humans have created AI, but AI is also reshaping the world we live in. With the explosion of large language model technology, AI has begun to further integrate into our lives, and humans need to consider and implement necessary security measures in the early stages of its development to avoid potential risks.

The cybersecurity problems AI encounters can be seen everywhere. When Tencent Technology interviewed Zhu Jun, deputy dean of the Institute of Artificial Intelligence of Tsinghua University, chief scientist of Beijing Zhiyuan Artificial Intelligence Research Institute and Ruilai Intelligence, he said ** "In fact, there is no network system that is always safe and unbreakable in the world. If the cost is not considered, criminals will use various methods to attack the system, and it is impossible to defend against it.” **

For more than 10 years, Zhu Jun has been committed to solving the problem of artificial intelligence security. He broke through the classic Bayesian basic theory and key technologies, proposed an efficient algorithm for the diffusion model, and incubated national-level specialized and new "little giant" enterprises through the transformation of achievements , Laying a solid foundation for the development of safe and reliable artificial intelligence.

The risks of AI itself cannot be ignored. Artificial intelligence is endowed with powerful learning and inferring capabilities, but this ability also leads to the high reliance of AI systems on data, which may reflect the bias of data sources in its decision-making and recommendations. A series of concerns, challenging the balance of AI ethics and justice.

When AI falls into a security crisis of trust, how should we respond? When the interactive application of AI and humans becomes more and more popular, how should we prevent potential safety hazards? In this dialogue, Zhu Jun talked about specific defense methods for improving the security and reliability of AI. We need to pay careful attention to exploring the moral and social implications of AI safety, and avoid being led into an unknown and uncontrolled future.

The following is the essence of the text, with deletions and adjustments on the premise of not changing your wishes:

There is no always safe network system

**Tencent Technology: You have been promoting research in the field of artificial intelligence against security. What security problems will the current application of AI technology bring? How should we identify these security issues? **

**Zhu Jun:**Artificial intelligence includes several key elements, such as data, algorithm model and application layer. In each element, we need to deal with various security issues in it.

At the data level, we need to pay attention to security issues such as data poisoning, data leakage, user privacy, and protection of core confidential data; at the model level, we need to deal with security issues such as confrontation algorithms. For example, when face recognition is used for unlocking, the attacker may unlock the verification system of the target mobile phone through a pair of specially-made anti-glasses (that is, "adversarial samples"), causing risks. In addition, if the model is maliciously implanted with a backdoor, the security of the model will also be threatened; at the application level, the security problem of artificial intelligence is also becoming more and more prominent. For example, deep synthesis, AIGC and other tools are used by criminals to create false content and engage in illegal purposes such as fraud and deception. These are all security issues faced by artificial intelligence in actual use or development.

For solutions and countermeasures, we need to use more advanced algorithms to automatically identify these contents, which is a hot and difficult issue in the field of artificial intelligence. However, this technology is like the relationship between "spear and shield", and the development of generative technology will promote the emergence of corresponding detection and defense technologies. At the same time, technologies on the generation and attack sides are constantly evolving. Due to the nature of the technology itself, no system is always secure and impossible to break. If the cost is not considered, criminals will use various methods to attack the system, which is hard to guard against.

Therefore, from the perspective of technology, **we need to deal with it in the form of "AI recognizes AI". But defense is actually more difficult than attack. Currently, we look for various mechanisms to improve the defense capabilities of the model, and take various protective measures when the model is used and deployed. For example, in the face recognition system, we deploy a face recognition firewall to detect and filter out suspicious or anti-attack samples before the samples enter the final recognition link, so as to achieve the purpose of protecting the system. At present, such technology has been implemented in banking and other industries.

**Tencent Technology: You mentioned that any network system has security loopholes. Currently ChatGPT has set off an upsurge in overseas applications. It has achieved good interaction. What kind of risks will there be? **

Zhu Jun: At present, large language models such as ChatGPT are in the process of rapid development, but at the same time, they also bring potential risks-for example, there will be some "injection attacks". From an algorithmic point of view, if someone injects specific words or symbols with ulterior motives, it may induce logic confusion and output errors in the large model.

In a multi-round dialogue system, it is difficult to defend against injection attacks. Hackers may perform injection attacks in various ways, and due to the large-scale model context understanding technology, the attack effect will be delayed, which is a new challenge for algorithm detection and defense. In this regard, we need to use a method similar to reinforcement learning to reverse the algorithm to detect and defend against words that may be maliciously injected. **The system can be used with peace of mind only if it is guaranteed that the system is not maliciously injected during the training process, or is not implanted with backdoors and other vulnerabilities.

From the perspective of the application level, there may also be some risks of malicious use of the dialogue system, such as hackers trying to bypass anti-injection protection measures to generate low-quality or bad content, including illegal information related to pornography and violence, which will become a part of the follow-up process. Issues that require independent detection and resolution.

**Tencent Technology: We just talked about the security issues of GPT, let’s take a closer look: what is the security defense capability of the server of GPT and other large models, and may it be attacked by hackers? **

Zhu Jun: Theoretically, it is completely possible. Because it is a large information system, any system will have loopholes. Therefore, in the process of system construction, we need to deploy various protection methods in advance as much as possible to improve the security of the system. Recently, we have also seen related cases: some attackers use ChatGPT to generate automatic attack codes, allowing it to find vulnerabilities in a target system more efficiently, and even further exploit vulnerabilities to launch attacks, so security problems will continue to exist.

Humans cannot accurately define and measure the intelligence level of AI

**Tencent Technology: In addition to the hidden dangers of hacker attacks, we are also worried about the security risks of AI itself. First of all, let's focus on a topic that everyone is currently discussing - do you think AI will produce consciousness? **

**Zhu Jun: My personal point of view is more inclined to think that the current performance of "consciousness" in artificial intelligence is not very clear, because we cannot accurately define and measure consciousness. ** Therefore, when observing the performance of the language model, we will find that the large model still has problems such as factual errors. While some of the errors read fluently, on closer inspection they are not factual or logical. This is one of the many problems with the model, that the level of specificity of consciousness it has has not been fully quantitatively assessed.

Language models are powerful learners because they know more about corpora and text than any human in the world. For example, a model might have access to nearly all the information available on the Internet, compared to the limited information resources each of us has access to.

From the perspective of versatility, AI is definitely better than any one person. However, in some respects, the performance of the model cannot reach the human level. Therefore, we should look at a model from the perspective of realistic technological development, including AGI and other aspects discussed by everyone. Personally, I think that the current level of technology has not reached the situation where it is out of control or only evolved under the control of the robot itself.

It can be said that large-scale machine learning models can use complex networks such as deep learning to process data, and draw on some human cognition in terms of architecture and design. But on the whole, there are huge differences between these artificial neural network models and real biological systems, ranging from scale to structure. Therefore, in fact, we currently cannot clearly evaluate the intelligence level of artificial intelligence systems, or assess whether it has cognitive abilities such as mind.

**Tencent Technology: Recently, some merchants launched the concept of "AI companion" - people can fall in love with AI, and they need to pay. Do you think AI can understand human emotions? What security risks exist in the process of interacting with virtual partners? **

Zhu Jun: Affective computing has always been a classic topic in the field of artificial intelligence. In terms of emotion, artificial intelligence technology can simulate a character and set its emotional or psychological state. However, from a technical point of view, there are still many problems and challenges in this field.

It is very difficult to achieve the level of true human communication. For example, even if we chat face to face or use the same language for dialogue, it is difficult to truly understand each other's emotions or mental activities, because each individual responds to the same input in thousands of ways. These large models we use now essentially model this process, but all modeling requires simplified and idealized assumptions. It is questionable whether these assumptions apply to everyone, or whether they fit well with each individual's reality. It is difficult for us to accurately express everyone's complex emotions with a simple model.

This model may involve various aspects such as social issues, ethics and morality, and there are many potential problems that need to be resolved. Although there are not many thresholds for technical implementation, and this model has already appeared in foreign countries. However, we need to think deeply about the impact of this model—for example, some young people may be less willing to spend energy on real love or marriage, etc. These may cause potential problems for social stability.

In addition, we need to pay attention to whether such artificial intelligence products will be biased or purposefully guided against certain individuals, which will bring great risks. If we interact with a robot every day, the information obtained will naturally be guided by the robot, which may affect personal values, or control personal emotions and behaviors. In the long run, this may affect the social relationship between people and cause changes in the behavior of the whole society. But these are not problems that can be solved entirely by technology. Generally speaking, compared with other countries, my country will be more cautious when using new technologies, and we will give early warning of possible risks and take some preventive measures.

Shaping Safe AI: Treating the Best Models as "Mentors"

**Tencent Technology: If there is an error in AI, from a technical perspective, what work can we do to correct the error in the large model? **

**Zhu Jun:**Because the training data and technical level are different, for example, we use the same question to ask different large models, the results they provide may be different, some results are good, but some are malicious or bad result. Therefore, it is necessary for us to standardize and improve the quality and controllability of these models.

Some large models usually do a lot of alignment and adversarial training. For example, before the advent of GPT-4, professionals in different fields asked questions from different angles to check the accuracy of the model to see if the system would produce non-compliant or malicious results, and try to regulate and adjust. However, there are still many models (including many open source models) that have not undergone such rigorous testing or adversarial training, so there will be various security risks.

One technical path worth trying is to treat one of the best models as a "mentor", and then force other models to mimic the behavior of this model in an efficient and economical way. Of course, there are more other aspects of work, such as normative and alignment work for each specific model according to the normative requirements of different countries.

While we expect these models to always produce spec-compliant results when used, the probability of risk never drops to zero. **In addition, when using it, we also need to consider ethics, legal rules, etc., which require joint management and regulation by different industries and fields, so that the model can better serve human beings.

**Tencent Technology: We just mentioned that through continuous training to correct and reduce the error rate of large models, how should we measure its reliability? You have been deeply involved in the field of Bayesian deep learning. In your opinion, how to build and optimize models to improve the accuracy and reliability of predictions? **

**Zhu Jun: **The industry has basically the same goal for accuracy, usually measured by objective indicators, and the specific indicators are related to the specific tasks performed. In terms of classification and recognition, the final recognition accuracy will be used to guide the training of the model.

For problems with uncertainty, such as neural networks, we found that in many cases, its predictions will be overconfident and optimistic. For example, the output of some results is originally a vague or uncertain prediction, but it will tell you the prediction result with overconfidence, which we call "overconfidence".

For this phenomenon or problem, deep learning techniques using Bayesian methods can better characterize uncertainty. It can mainly be considered from many aspects, such as the uncertain factors that may exist at the input end and the uncertain factors that may exist at the model end, and give a confidence that is more in line with the actual situation. This Bayesian approach is more reliable than neural networks.

**Tencent Technology: The network structure in the real world is often very complex, including multi-level, multi-dimensional, dynamic changes and other characteristics, which will bring great challenges to the establishment and optimization of the diffusion probability model. The team you lead is one of the earliest teams engaged in the research of diffusion probability model theory and algorithm in the world. How does your team eliminate noise and data uncertainty in model construction to improve the robustness and reliability of the model? of? **

Zhu Jun: Diffusion model is a generative model, which has two processes of forward diffusion and reverse diffusion. Forward diffusion turns an image into a completely random Gaussian noise image by gradually adding noise. The reverse diffusion starts from a distribution with almost no structure, gradually denoises, and converges to a distribution that can describe real data. New samples can be generated from this distribution, such as text, picture and video generation, which are widely studied now.

Diffusion models are one of the most critical techniques in the generative field. In terms of robustness, the idea of diffusion models is similar to adversarial examples. Adversarial examples achieve the purpose of attack by adding algorithm-optimized noise in the generation process. In turn, we can optimize the magnitude and direction of the noise by gradually finding the distribution in the reverse diffusion process to improve the robustness of the model. This method can also be applied to the generation of noisy data to improve the reliability and accuracy of the model.

**Tencent Technology: How can we improve the accuracy of AI in the application of Vincent in other directions? I am concerned about the new Wensheng 3D algorithm ProlificDreamer recently proposed by your team, which can generate ultra-high-quality 3D content without any 3D data. How does your team deal with semantic diversity and ambiguity to generate more accurate content? 3D model? **

Zhu Jun: Compared with traditional 3D methods, the industry usually uses a 2D pre-trained generative model (such as a diffusion model) to train on an image database. When doing 3D generation, we need to map the generated 2D image onto the 3D model, which requires an intermediate step called "distillation". Since the 3D model has a spatial structure, we need to consider the 3D properties of the object. Therefore, we need to observe objects from various angles and render the corresponding 2D images, and then align them to the pre-trained model, so that 3D assets can be generated, etc. However, this approach also has some limitations. For example, the results it generates are usually too saturated or too smooth, lacking information such as details and textures.

In order to solve this problem, we need to explore lower-level technologies. We found that there are some inherent difficulties in using the distillation algorithm to find a single 3D model, which needs to be overcome from the basic principles. Existing algorithms look for some kind of extremum in the objective function, similar to the "greedy algorithm (Greedy algorithm)", it will only find the optimal solution, in order to achieve this purpose, existing work changes the objective function to make it higher in some areas , which is more average in other regions, this method of objective function adjustment can quickly find the final solution.

To overcome the difficulties of the above approaches, we reformulate the text-to-3D generation problem as sampling from some distribution that the 3D model might obey, and then rendering and aligning it with the pre-trained 2D model . **The advantage of this sampling method is that the 2D model itself is a probabilistic model, and the description information is richer than greedy optimization; for this reason, we derived a new variational distillation algorithm and used it in basically the same Many very detailed and complex 3D scenes, including high-resolution assets, were generated in a fraction of the time.

The key point of our method is that it reduces or completely removes the dependence on 3D training data and significantly improves the quality of generation. Recently, I communicated with practitioners who do graphics, and they also feel that this effect is quite amazing. Let us see the great potential of being able to generate high-quality 3D images.

For the handling of ambiguity. For the same text input, different people may have different understandings, for example, the word "Apple" may refer to Apple, Apple Inc. or its products. In our model, the ambiguity is resolved by using a probability-based sampling approach to generate multiple possible outcomes. ** In the long run, disambiguation requires more cues and alignments to improve controllability and precision, such as the alignment of text and images or other modal data. **In language and multimodal domains, the ultimate meaning is relative to the relevant context.

Currently, we are working with customers in different industries to further improve our 3D generation technology and make it more mature. In the 3D field, high-quality 3D assets are of high value. For example, in game asset creation scenarios, traditional companies usually use traditional methods such as geometry or graphics to build and maintain 3D asset libraries, which requires a large time investment. Our technology It can greatly improve the creative efficiency and reduce the time cost.