January 12, 2023

WithSecure security researchers demonstrated how the GPT-3 natural language generation model and the ChatGPT can be used to carry out phishing scams.

WithSecure ran a number of experiments to see how modifying the language model’s input influenced the text output. These included phishing and spear-phishing, harassment, social validation for scams, the appropriation of a written style, the creation of intentionally divisive opinions, using the models to create prompts for malicious text, and fake news.

GPT-3 even created a convincing email thread to use in a phishing campaign, as well as social media posts with hashtags to harass a fictitious CEO of a robotics company. It made use of prompt engineering, a concept related to large language models that entails discovering inputs that produce desirable or useful results in order to generate a variety of content deemed harmful by the researchers.

It also investigated how changes in inputs to the existing models affected the synthetic text output. The goal was to identify how AI-language generation can be abused through malicious and creative prompt engineering, with the hope that the findings will help guide the development of safer large language models in the future.

The sources for this piece include an article in TheRegister.