3 ms·
I'm a bit surprised at all the negative comments here. I hope it isn't too discouraging for your team, because as author of a 50K LOC Python app (HashBackup),
by prirun 7y ago
I'm a bit surprised at all the negative comments here. I hope it isn't too discouraging for your team, because as author of a 50K LOC Python app (HashBackup), I could really use this! I love the Python language but sometimes the performance is a drag. For example, to plan a restore when the data isn't local, HashBackup has to traverse every block in every file to be restored and figure out when to load the block and when it can be released from the cache. This isn't particularly difficult, but for very large restores it requires long loops using large lists, arrays, and/or dicts. Parts are coded in Cython, and that works well for easily-isolated functions, but not so great for something like the restore plan that needs database access and is referenced during the restore.
I ran a small 10M entry {int:int} dict benchmark. In Python 2.7, the test used 1.1GB of RAM and about 8 seconds. In D (fully compiled) the same test used 881MB and 7.4 seconds. Here's the D version:
import std.stdio : writeln;
int main() {
int[int] map;
foreach (i; 0..10_000_000)
map[i] = i;
foreach (i; 0..3) {
foreach (j; 0..10_000_000) {
map[j] = map[j] + 1;
}
}
return 0;
}
In Seq it ran in 5 seconds and used 395MB. Here's the test program and Seq run:
map = dict[int,int]()
for i in range(0,10000000):
map[i] = i
for i in range(3):
for j in range(10000000):
map[j] = map[j] + 1
print map[12345]
[root@hbseq ~]# /usr/bin/time -v ./map
12348
User time (seconds): 4.67
System time (seconds): 0.55
Percent of CPU this job got: 97%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.36
Maximum resident set size (kbytes): 395532
Looks pretty great to me, especially if I don't have to do a major rewrite! I guess I could have hit a case where Seq happened to have a higher HT load for 10M entries and D just did a resize, so it would be good to run the same kind of test at a lot of different hash table sizes. But the Python results are pretty terrible space-wise.