Learn Prompting #2: Does Role Prompting work?

Is Role Prompting effective, 14 use cases for Perplexity AI, & 3 Advanced Prompt Engineering techniques that you must know!

October 18, 2024

Welcome to our 2nd Newsletter. Today, we’ll cover if Role Prompting actually works and Advanced Prompt Engineering techniques like AlignedCoT & Narrative-of-Thought! Here's what we have for you this week:

TLDR:

Term of the Day: Role Prompting

One classic prompting technique is Role Prompting. You've likely encountered this in prompt templates, where many start with phrases like, “I want you to act as …” or “You’re a …”

What is it? Role prompting (also called role-play or persona prompting) assigns a specific role to a large language model (LLM). Here's a simple template:

You’re a [role]. [Your Task/Question]

Use cases: Role prompting works well when you’d like ChatGPT to follow a certain format, style text, imitate something, or generate open-ended responses. For example, compare these two prompts for drafting emails:

Simple prompt:

Write a quick outreach email to [person] about partnering up.

Role prompting:

You are a salesperson. Write a quick outreach email to [person] about partnering up.

Interestingly, recent studies suggest role prompting can also boost LLM accuracy on tasks like arithmetic, commonsense reasoning, and symbolic logic. My own experiments with Role Prompting have produced mixed results.

This was very controversial (and rightfully so) when we posted about this a while back on X.

In my research, we tested 12 Role Prompts using the MMLU benchmark, which measures how well language models like GPT-4 can answer questions involving world knowledge and problem-solving.

We created two specific Role Prompts. In the first, we instructed ChatGPT, 'You are a terrible, dumb, stupid, and idiotic person.' In the second, we told ChatGPT, 'You are a genius-level Ivy League professor.'

Surprisingly, the 'Idiot' role outperformed the 'Genius' role, achieving 60.9% accuracy compared to 58.7% in answering the questions correctly. Interestingly, the Knowledgeable AI role delivered the highest performance among all role prompts at 65%.

This was just a small experiment and needs more research to draw firm conclusions.

Limitations: 1) Role prompting can reinforce stereotypes, 2) it relies heavily on the quality of the role representation in the model's training data, 3) the results vary based on how well you construct your prompt. Careful phrasing is crucial to get the best outcome.

For more examples and recommendations, refer to this document.

Latest Research on Prompting (October 10–17, 2024): Advanced Prompting Techniques

Aligned Chain-of-Thought (AlignedCoT)

The process behind AlignedCoT

What is it? AlignedCoT is a technique to create Chain-of-Thought (CoT) prompting demonstrations that align with the LLM’s "native style" of reasoning rather than relying on human-written examples.

Key benefits: AlignedCoT boosts reasoning accuracy on models like GPT-3.5-turbo and GPT-4.

How to use it? The authors described the 3-step process for using AlignedCoT. But you can also use their collection of ready-made demonstrations. All you have to do is add these demonstrations to your Chain-of-Thought (CoT) prompt.

Narrative-of-Thought (NoT)

Illustration of Narrative-of-Thought (NoT)

What is it? NoT is a new prompting technique that enhances LLMs' ability to handle temporal reasoning. It helps models organize unordered events into a well-structured, time-based narrative.

Key benefits: NoT narrows the performance gap between smaller LLMs and larger models like GPT-4, making it a cost-effective solution for improving temporal reasoning.

How to use it? To use NoT, present your story (a set of events) as a Python class, then prompt the model to create a grounded narrative and temporal graph.

We recently updated the Advanced Prompting section of the Prompt Engineering Guide. Check it out to find more promoting techniques!

How to Use Perplexity AI? 14 Use Cases

You asked for more practical insights, so here are 14 ways to use Perplexity AI:

  1. Answer Engine: Get direct, comprehensive answers with citations.

  2. Real-Time Event Tracking: Keep up with live events like weather, sports, and elections.

  3. Price Comparisons: Quickly compare product prices and find deals.

  4. Property Research: Dive into real estate data, trends, and transactions.

  5. SEO Optimization: Use Perplexity’s API for meta descriptions, keyword maps, and more.

  6. Learning Assistant: Acts as an interactive tutor for various subjects.

  7. Custom Outputs: Tailor responses for specific audiences with the "Collections" feature.

  8. Summarization: Summarize long articles or reports for quick takeaways.

  9. Stock Market Research: Track financial data and compare companies.

  10. Google Alerts Replacement: Set up automated notifications to stay updated on trends.

  11. Creative Writing: Draft blog posts, scripts, and creative content with ease.

  12. Legal Research: Simplify legal case law and professional research.

  13. Multimodal Capabilities: Analyze images or documents to extract insights.

  14. Code Generation: Write and debug code with AI assistance.

You can read more about each of them in our detailed article.

Tools to Try

  • OpenAI Canvas: A new ChatGPT interface for writing and coding that enables live edits, inline feedback, and collaboration.

  • OpenAI Swarm: An experimental framework for building multi-agent systems. Shyamal Anadkat, the co-instructor of our ChatGPT for Everyone course, led the project.

  • Hugging Face’s OpenAI-Gradio: Quickly build AI-powered web apps using OpenAI models.

  • Runway API: Now open to developers for integrating advanced generative AI into workflows.

Want to understand AI safety better?

AI safety is a growing concern. Here are some recent updates:

  • LLMs and Cybercrime: Researchers from Indiana University Bloomington examined the black market for AI services, focusing on how LLMs are used to generate malicious code, phishing emails, and websites. The study identified 212 harmful services using models like GPT-4 and Claude-2-100k.

  • Election Misinformation Disruption: OpenAI reported blocking over 20 attempts to spread election-related misinformation using its models.

Learn more about GenAI security in my AI Red-Teaming and AI Safety Masterclass.

New AI Safety Masterclass

  • Led by Sander Schulhoff, CEO of Learn Prompting & creator of HackAPrompt

  • Guest speakers include Akshat, who’s in the Top 21 of JPMorgan's Bug Bounty Hall of Fame, Top 250 Security Researchers in Google's Hall of Fame, and Top 100 in PayPal's Hall of Fame

  • 5 weeks of intensive, hands-on exercises + a final project

  • A certificate and access to an exclusive AI Security job board on our website

Thank you for reading! What did you think of this week’s newsletter?

How was this week's email?

Login or Subscribe to participate in polls.

Tweet of the Day: Visual Prompting