3 ms·

A couple of tests I've done in their live demo. - Implement the quicksort algorithm in python. I like this test because usually LLMs do the first time using li

A couple of tests I've done in their live demo. - Implement the quicksort algorithm in python. I like this test because usually LLMs do the first time using list comprehensions, then I ask to avoid allocations. - Write the game "snake" in python, and then in JS/HTML. It succeeded on the simple quicksort implementation with list comprehensions, but failed when I asked for the in-place implementation. With the snake game tests, it didn't get even close. In my experience these results are worse than non-code top 7B LLMs such as OpenChat/NeuralHermes, and significantly worse than plain Deepseek-coder-instruct 6.7B (which is the only small model I've tested that gets the "snake" game to work). When I ran these tests locally I used Q6 quantization (GGUF), and I assume their live demo is not quantized. I find the benchmark results surprising, to the point I wonder if the benchmarks didn't leak into the training set.

UniverseFly 3y ago
Thanks for testing Magicoder! I tried “Implement an in place quicksort” with temperature 0 using the demo and here is the output I get. I haven’t checked every detail but it looks correct: Here is a Python implementation of an in-place quicksort: def partition(arr, low, high): i = (low-1) pivot = arr[high] for j in range(low, high): if arr[j] <= pivot: i = i+1 arr[i], arr[j] = arr[j], arr[i] arr[i+1], arr[high] = arr[high], arr[i+1] return (i+1) def quickSort(arr, low, high): if len(arr) == 1: return arr if low < high: pi = partition(arr, low, high) quickSort(arr, low, pi-1) quickSort(arr, pi+1, high) arr = [10, 7, 8, 9, 1, 5] n = len(arr) quickSort(arr, 0, n-1) print("Sorted array is:", arr) This code first defines a partition function that takes an array and two indices, low and high, and rearranges the elements of the array so that all elements less than the pivot (the last element in the array) come before it, and all elements greater than the pivot come after it. Then it defines a quickSort function that takes an array and two indices, low and high, and sorts the array in-place using the partition function. If the array has more than one element, it recursively sorts the subarrays to the left and right of the pivot. Finally, it tests the function with an array of integers.
RedXenon 3y ago
I used the prompt "Write snake game." on the live demo with temperature = 0 and max tokens = 2048. I got the correct implementation of the snake game in Python. But even after a few tries, I could not generate a snake game in HTML/JS.

UniverseFly 3y ago

Thanks for testing Magicoder! I tried “Implement an in place quicksort” with temperature 0 using the demo and here is the output I get. I haven’t checked every detail but it looks correct: Here is a Python implementation of an in-place quicksort: def partition(arr, low, high): i = (low-1) pivot = arr[high] for j in range(low, high): if arr[j] <= pivot: i = i+1 arr[i], arr[j] = arr[j], arr[i] arr[i+1], arr[high] = arr[high], arr[i+1] return (i+1) def quickSort(arr, low, high): if len(arr) == 1: return arr if low < high: pi = partition(arr, low, high) quickSort(arr, low, pi-1) quickSort(arr, pi+1, high) arr = [10, 7, 8, 9, 1, 5] n = len(arr) quickSort(arr, 0, n-1) print("Sorted array is:", arr) This code first defines a partition function that takes an array and two indices, low and high, and rearranges the elements of the array so that all elements less than the pivot (the last element in the array) come before it, and all elements greater than the pivot come after it. Then it defines a quickSort function that takes an array and two indices, low and high, and sorts the array in-place using the partition function. If the array has more than one element, it recursively sorts the subarrays to the left and right of the pivot. Finally, it tests the function with an array of integers.

RedXenon 3y ago

I used the prompt "Write snake game." on the live demo with temperature = 0 and max tokens = 2048. I got the correct implementation of the snake game in Python. But even after a few tries, I could not generate a snake game in HTML/JS.