6 ms·

Counter is not in Python's prelude. Therefore you have to import it in order to use it. My comment about prelude was in response to you saying I was using Pyth

Counter is not in Python's prelude. Therefore you have to import it in order to use it. My comment about prelude was in response to you saying I was using Python's “language+plus-its-entire-ecosystem”. This is not true; Counter is part of the standard library that comes with the language. It is not from a third-party package. > If it takes that many keystrokes, you're definitely only going to use it once per year! This is the normal way you make enumerations in q/k. You don't use this frequently in other languages because you almost never care about the index of a value in a hash table (because it's ridiculous), and you almost never want to brute-force unique a list (because it's a footgun).

geocar 7y ago
> you saying I was using Python's “language+plus-its-entire-ecosystem”. This is not true; Counter is part of the standard library that comes with the language. It is not from a third-party package. I'm not sure I agree "third-party package" is a good/useful place to draw the line, but I don't think it's terribly important. Sorry. > you almost never care about the index of a value in a hash table (because it's ridiculous) So I want to query all of the referrer urls I've seen recently. I only see a few of them, so I want to have a table of all the URLs, and another table with one row per event (landing on my page). In SQL, you might have a foreign-key operation, but in q, I can also make the enumeration directly. I don't really care about the value of the index, just that I have one.
- Veedrac 7y ago
  I wouldn't want to use an index as a long-lived key like that, since it makes ops like deletion a footgun. For temporary calculations I'd just `{u: Url(u) for u in url_strings}` for the table of URLs, and stick the `Url` objects in the table of events.
  - geocar 7y ago
    I don’t understand any of this. I don’t know what footgun is. I have no idea what a “URL” object is or how you stick it in a table of events (on disk? in memory? in a remote process called a “database”). I’m don’t know what you mean by temporary calculations. I have fifty billion of these events to handle every day, and I do this on one server (that is actually doing other things as well). It is not an option to “just” do anything at this scale if I want to keep costs under control.
    - Veedrac 7y ago
      Sorry, it's a bit weird not sharing a lexicon. I get the impression Q is a whole different genealogy of programmers. In Python, although you can just use shelve to store data on disk, in practice this is considered a bad idea beyond very simple cases. Valuable data wants the guarantees that real databases provide, like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events a day, live, and you need these to persist, you'd use SQL or something similar. That would then ultimately determine how you add and manipulate records. If you don't care that much if you lose the data on a crash, that's when we're talking about temporary calculations. In Python, rather than having two tables like (eg.) URLs: url_id url count 1 google.com 300 2 python.org 400 3 example.net 200 Requests: request_id data url_id 1 'spam' 2 2 'eggs' 2 ... 900 'ham' 1 you would make a custom type, aka. ‘Url’, containing ‘url’ and ‘count’ as object fields, and then store your requests as a list containing references to those ’Url’s. urls = { "google.com": Url("google.com", 300), "python.org": Url("python.org", 400), "example.net": Url("example.net", 200), } requests = [ Request("spam", urls["python.org"]), Request("eggs", urls["python.org"]), ..., Request("ham", urls["google.com"]), ]
      - geocar 7y ago
        > ... like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events a day, live, and you need these to persist, you'd use SQL or something similar. That would then ultimately determine how you add and manipulate records. … ACID is overrated. You can get atomicity, consistency, isolation and durability easily with kdb as I'll illustrate. I appreciate you won't understand everything I am saying though, so I hope you'll be able to ask a few questions and get the gist. First, I start write my program in g.q and start a logged process: q g -L This process receives every event in a function like this: upd:{r[`u?y`url;y`metric]+:1} There's my enumeration, saved in the variable "u". "r" is a keyed table where the keys are that enumeration, and the metric is whatever metric I'm tracking. I checkpoint daily: eod:{.Q.dd[p:.Q.dd[`:.;.z.d];`r] set r;.Q.dd[p;`u] set u;r::0#r;system"l"} This creates a directory structure where I have one directory per date, e.g. 2020.03.11 which has a file (u or r) referring to the snapshots I took. I truncate my keyed table (since it's a new day), and then I tell q the logfile can be truncated and processing continues! To look at an (emptyish) tree right after a forced checkpoint: total 24 drwxr-xr-x 4 geocar staff 128 11 Mar 16:59 2020.03.11 -rw-r--r-- 1 geocar staff 8 11 Mar 16:59 g.log -rw-r--r-- 1 geocar staff 206 11 Mar 16:55 g.q -rw-r--r-- 1 geocar staff 130 11 Mar 16:59 g.qdb geocar@gcmba a % ls -l 2020.03.11 total 16 -rw-r--r-- 1 geocar staff 120 11 Mar 16:59 r -rw-r--r-- 1 geocar staff 31 11 Mar 16:59 u The g.q file is the source code we've been exploring, but the rest are binary files in q's "native format" (it's basically the same as in memory; that's why q can get this data with mmap). If I've made a mistake and something crashes, I can edit g.q and restart it, the log replays, no data is lost. If I want to do some testing, I can copy g.log off the server, and load it into my local process running on my laptop. This can be really helpful! I can kill the process, turn off the server, add more disks in it, turn it back on, and resume the process from the last checkpoint. You can see some of these qualities are things only databases seem to have, and it's for that reason that kdb is marketed as a database. But that's just because management has a hard time thinking of SQL as a programming language (even though it's there in the name! I can't fault them, it is a pretty horrible language), and while nobody wants to write stored procedures in SQL, that's one way to think about how your entire application is built in q. That's basically it. There's a little bit more code to load state from the checkpoint and set up the initial days' schema for r: u:@[get;.Q.dd[.Q.dd[`:.;last key `:.];`u];{0#`}]; r:([url:`u$()]; req:0#0; imp:0#0; clk:0#0; vt:0#0); but there's no material difference between "temporary calculations" or ones that will later become permanent: All of my input was in the event log, I just have to decide how to process it.

geocar 7y ago

> you saying I was using Python's “language+plus-its-entire-ecosystem”. This is not true; Counter is part of the standard library that comes with the language. It is not from a third-party package. I'm not sure I agree "third-party package" is a good/useful place to draw the line, but I don't think it's terribly important. Sorry. > you almost never care about the index of a value in a hash table (because it's ridiculous) So I want to query all of the referrer urls I've seen recently. I only see a few of them, so I want to have a table of all the URLs, and another table with one row per event (landing on my page). In SQL, you might have a foreign-key operation, but in q, I can also make the enumeration directly. I don't really care about the value of the index, just that I have one.

Veedrac 7y ago
I wouldn't want to use an index as a long-lived key like that, since it makes ops like deletion a footgun. For temporary calculations I'd just `{u: Url(u) for u in url_strings}` for the table of URLs, and stick the `Url` objects in the table of events.
- geocar 7y ago
  I don’t understand any of this. I don’t know what footgun is. I have no idea what a “URL” object is or how you stick it in a table of events (on disk? in memory? in a remote process called a “database”). I’m don’t know what you mean by temporary calculations. I have fifty billion of these events to handle every day, and I do this on one server (that is actually doing other things as well). It is not an option to “just” do anything at this scale if I want to keep costs under control.
  - Veedrac 7y ago
    Sorry, it's a bit weird not sharing a lexicon. I get the impression Q is a whole different genealogy of programmers. In Python, although you can just use shelve to store data on disk, in practice this is considered a bad idea beyond very simple cases. Valuable data wants the guarantees that real databases provide, like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events a day, live, and you need these to persist, you'd use SQL or something similar. That would then ultimately determine how you add and manipulate records. If you don't care that much if you lose the data on a crash, that's when we're talking about temporary calculations. In Python, rather than having two tables like (eg.) URLs: url_id url count 1 google.com 300 2 python.org 400 3 example.net 200 Requests: request_id data url_id 1 'spam' 2 2 'eggs' 2 ... 900 'ham' 1 you would make a custom type, aka. ‘Url’, containing ‘url’ and ‘count’ as object fields, and then store your requests as a list containing references to those ’Url’s. urls = { "google.com": Url("google.com", 300), "python.org": Url("python.org", 400), "example.net": Url("example.net", 200), } requests = [ Request("spam", urls["python.org"]), Request("eggs", urls["python.org"]), ..., Request("ham", urls["google.com"]), ]
    - geocar 7y ago
      > ... like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events a day, live, and you need these to persist, you'd use SQL or something similar. That would then ultimately determine how you add and manipulate records. … ACID is overrated. You can get atomicity, consistency, isolation and durability easily with kdb as I'll illustrate. I appreciate you won't understand everything I am saying though, so I hope you'll be able to ask a few questions and get the gist. First, I start write my program in g.q and start a logged process: q g -L This process receives every event in a function like this: upd:{r[`u?y`url;y`metric]+:1} There's my enumeration, saved in the variable "u". "r" is a keyed table where the keys are that enumeration, and the metric is whatever metric I'm tracking. I checkpoint daily: eod:{.Q.dd[p:.Q.dd[`:.;.z.d];`r] set r;.Q.dd[p;`u] set u;r::0#r;system"l"} This creates a directory structure where I have one directory per date, e.g. 2020.03.11 which has a file (u or r) referring to the snapshots I took. I truncate my keyed table (since it's a new day), and then I tell q the logfile can be truncated and processing continues! To look at an (emptyish) tree right after a forced checkpoint: total 24 drwxr-xr-x 4 geocar staff 128 11 Mar 16:59 2020.03.11 -rw-r--r-- 1 geocar staff 8 11 Mar 16:59 g.log -rw-r--r-- 1 geocar staff 206 11 Mar 16:55 g.q -rw-r--r-- 1 geocar staff 130 11 Mar 16:59 g.qdb geocar@gcmba a % ls -l 2020.03.11 total 16 -rw-r--r-- 1 geocar staff 120 11 Mar 16:59 r -rw-r--r-- 1 geocar staff 31 11 Mar 16:59 u The g.q file is the source code we've been exploring, but the rest are binary files in q's "native format" (it's basically the same as in memory; that's why q can get this data with mmap). If I've made a mistake and something crashes, I can edit g.q and restart it, the log replays, no data is lost. If I want to do some testing, I can copy g.log off the server, and load it into my local process running on my laptop. This can be really helpful! I can kill the process, turn off the server, add more disks in it, turn it back on, and resume the process from the last checkpoint. You can see some of these qualities are things only databases seem to have, and it's for that reason that kdb is marketed as a database. But that's just because management has a hard time thinking of SQL as a programming language (even though it's there in the name! I can't fault them, it is a pretty horrible language), and while nobody wants to write stored procedures in SQL, that's one way to think about how your entire application is built in q. That's basically it. There's a little bit more code to load state from the checkpoint and set up the initial days' schema for r: u:@[get;.Q.dd[.Q.dd[`:.;last key `:.];`u];{0#`}]; r:([url:`u$()]; req:0#0; imp:0#0; clk:0#0; vt:0#0); but there's no material difference between "temporary calculations" or ones that will later become permanent: All of my input was in the event log, I just have to decide how to process it.