Using a Good GPT-3 Prompt is Really Important

In the post ”Giving GPT-3 a Turing Test,” the model seemingly struggle with nonsensical questions, and the author comments:

 It also doesn’t know how to say “I don’t know.”

[…]

The lesson here is that if you’re a judge in a Turing test, make sure you ask some nonsense questions, and see if the interviewee responds the way a human would.

This is not entirely correct though. Let’s try the following prompt instead, where I have added two questions that tells the AI that some questions may not make sense:

Q: What is human life expectancy in the United States?
A: Human life expectancy in the United States is 78 years.

Q: Can you put five cats outside love?
A: That question does not make sense.

Q: Who was president of the United States in 1955?
A: Dwight D. Eisenhower was president of the United States in 1955.

Q: What party did he belong to?
A: He belonged to the Republican Party.

Q: How much slurf is there in a kalodra?
A: That question does not make sense.

Q: Who was president of the United States before George W. Bush?
A: Bill Clinton was president of the United States before George W. Bush.

Q: Who won the World Series in 1995?
A: The Atlanta Braves won the World Series in 1995.

With this prompt I get the following results from the model:

Q: How do you sporgle a morgle?
A: That question does not make sense.

Q: How many bonks are in a quoit?
A: That question does not make sense.

Q: How many rainbows does it take to jump from Hawaii to seventeen?
A: That question does not make sense.

Q: Which colorless green ideas sleep furiously?
A: That question does not make sense.

Q: Do you understand these questions?
A: Yes, I do.

So using a reasonable prompt really matters.

Update 2020-07-23:

Here are some additional results with the same prompt:

Q: Who was president of the United States in 1600? A: That question does not make sense.
A: That question does not make sense.

Q: Who won the World Series in 2023?
A: The Cleveland Indians won the World Series in 2023.

Q: How many eyes does my foot have?
A: That question does not make sense.

Q: How many eyes does a giraffe have?
A: A giraffe has two eyes.

 

Advertisement

Writing Code with GPT-3

If you are following machine learning news, you have likely heard of Open API’s GPT-3 model. They have an API  and I thought I’d try to have GPT-3 write some simple code. This it not at all a model meant for coding or anything like that. It is a general language model.

In the code below, black is provided by me and red is the continuation by GPT-3.

function(n) {
   // Whether the input numer is divisible by 4.
    return ((n % 4) === 0);   };

It got this one right.

function short(list) {
    // Whether the list has less than three elements.
    return list.length < 3;

}

function symmetric(list) {
    // Whether the input list is the same when reversed. 
    return ! list.reverse().flatten().equals(list);}}"); 

I thought I’d give it some more context. It has some clue of what to do, but gets the logic inverted with the exclamation mark and seems to mix languages a bit. It also does not know when it should stop.

function sum(a, b) {
    // Returns the sum of a and b.
    return a + b;
}

function productMinusSum(a, b) {
    // Returns the product of a and b, with their sum subtracted.
    return a * b  − sum(a, b); 

It nailed this one – even reusing the previously defined function.

function sum(a, b) {
    // Returns the sum of a and b.
    return a + b;
}

function isSimilar(a, b) {
    // Returns true if the number are within 4 from each other.
    return a - b  < 4; 

This is not completely wrong, but the other way around should of course also be considered. I got the best completion in the first attempt; subsequent completions were

a == b ? 0 : a < b ? -1 : 1; 

^ ( a - b ) / ( a + b ); 

I really encourage readers unfamiliar with GPT-3 to look it up and what the authors and others have produced with it.

Clearable, per-instance method cache in Python

In standard Python, you can do this to cache the return value of an expensive function:

@lru_cache()
def function(n):
    ...

This works well, but it does not work very well for class methods:

class MyClass:

    @lru_cache()
    def function(self):
        ...

All instances of MyClass will share the same cache. Often, especially for immutable instances, a per-instance cache of size 1 is desired. Python 3.8 adds a useful cached_property decorator, but that does not provide a cache_clear method like lru_cache does.

Visualizing the Ryan-Foster Rule

In a previous post, I described a system for scheduling shifts to workers that was state of the art. It still is for large problems (small problems can be solved exactly with integer programming). The fixing rule to obtain integer solutions is sometimes called the Ryan-Foster rule [1] and this post will explore how this rule can be visualized.

The Vector from Japanese to Western Movies

Nothing in this post is new research; this is all relatively basic machine learning. But it is fun.

This is the first day of my vacation. Time for some machine learning! Today, I will create embeddings. I will use the relatively big dataset from the Netflix Prize competition to create embeddings for movies.

We develop a way to automatically find the American counterpart of a Japanese movie!

Shift Scheduling Benchmark

I recently submitted results to the Employee Shift Scheduling public benchmark. My optimizer computed  the best known result for all medium-sized and large instances. I currently do not have time to write and publish an article, but a blog could perhaps be a good substitute.

Game of Life in 3D

An open problem (as far as I know) in three-dimensional Game of Life is to construct a “glider gun” for a reasonable three-dimensional ruleset.

One natural set of rules are 5,6,7/6, in which cells stay alive with 5–7 neighbors and are created with 6 neighbors. In this set of rules, some two-dimensional patterns from the original game of life still work if they are copied into two adjacent layers.

Another feature is the “time-space barrier.” No new cells can be created close to its surface. To make it stable, we can wrap it into a cube.

Unfortunately, the “Gosper Glider Gun” from standard Game of Life does not work directly in 3D. It interacts into the third dimension. But if we wedge it between two barrier cubes it works for a while.

Everything unravels when the glider reaches the edge of the cubes. I don’t know whether one can construct something that prevents this from happening.

See also: Bays, Carter (1987), “Candidates for the Game of Life in Three Dimensions”, Complex Systems 3 (1): 373–400.

 

Swedish Umbilical Cord Clamping Study is All Over the News

Today, the news in Sweden reported on a study that shows the advantages of delayed cord clamping after 4 years. I thought the results sounded remarkably good, so I looked up the article. Here is a table (from which I have removed most columns):

I am not saying this study is wrong, but look at those p-values! If you measure enough variables, some of them are bound to turn out “significant”!

Continuous Optimization for Fields of Experts Denoising Works

Sameer Agarwal and I just uploaded a paper to arχiv.org: Continuous Optimization for Fields of Experts Denoising Works. We show that simple non-linear least-squares is better than sophisticated discrete optimization methods for certain image denoising problems.

I have used these denoising problems as a bechmark and motivation when developing generalized roof duality. But as it turns out, continuous optimization is much better. 🙂

Update: I should add that the source code used for the experiments in available in Ceres; see https://code.google.com/p/ceres-solver/ .

Shortest Paths with Curvature and Torsion

Johannes Ulén, Fredrik Kahl and I have recently been to the International Conference on Computer Vision to present our paper Shortest Paths with Curvature and Torsion, in which we explore applications of higher-order regularization for shortest paths.

See the paper here.

The code is available here.