3 ms·
Would be interesting if this can do HTTP/1.1 pipelining. Need to take a closer look. What makes me hesitate to look closer is that I almost always am doing so
by 1vuio0pswjnm7 4y ago
Would be interesting if this can do HTTP/1.1 pipelining. Need to take a closer look.
What makes me hesitate to look closer is that I almost always am doing some text processing after the HTTP response and before the SQL commands. Due to resource constraints, I do not want to store HTML cruft.
I mostly am doing
HTTP response --> text processing --> SQLite3
rather than
HTTP response --> SQLite3
However I also have a need for storing different combinations of HTTP request headers and values in a database. I currently use the UNIX filesystem as a database (one header per file, djb's envdir loads the headers into environment from a selected folder), but maybe I could use SQLite3. For unprocessed text, e.g., from pipelined DoH responses, I use tmux buffers as a temporary database. Then do something like
HTTP responses --> tmux loadb /dev/stdin
tmux saveb -b b1 /dev/stdout|textprocessingutility --> ip-map.txt (append unique, i.e., "add unique")
The ip-map.txt file gets loaded into memory of a localhost forward proxy.
Databases, such as NetBSD's db, djb's cdb, kdb+, or sqlite3 can help with the "append unique" step if the data gets big.
Note: Any JSON I retrieve is "text", not binary data. Most HTTP responses are HTML, sometimes with embedded JSON. Pipelined DoH is binary but I still use drill to print the packets as text. (When I finally learn ldns I will stop using drill.)
- 1vuio0pswjnm7 4y agodjbdns instead of ldns Look also at possibility of libpcap for printing catenated DNS packets.
- alexgarcia-xyz 4y agore pipelining: unless Go's network package does this magically under the hood, then sqlite-http doesn't take advantage of that! If you think that would be a nice feature, file an issue and I'll see what can be done How complex is your text processing? If I'm working with a small dataset, I typically just save the "raw" responses into a table and create several intermediate tables that extracts the data I need. Something like: create table raw_responses as http_get('https:/....'); create table intermediate1 as select xxxx(response_body) as message from raw_responses; create table final as select message from intermediate; (some CTEs[0] will be very useful here) The "lambda" pattern described about half-way through this post might be useful as well, if you need to do some complex text processing. [0] https://www.sqlite.org/lang_with.html https://www.sqlite.org/lang_with.html
- robmccoll 4y agoGo's HTTP client doesn't support pipelining in HTTP 1.1. It will do keep alive / socket re-use. HTTP 2+ have support for request multiplexing which Go supports. Your code appears to create a new client on each request, so I'm not sure if the underlying transport will be re-used or if any multiplexing will take place. I would be surprised if it is.
- iknownothow 4y agoWouldn't you be doing JSON processing instead of text processing? I myself never had to do text processing but instead JSON processing and that was convenient to do in Postgres directly, and I assume it's convenient to do in SQLite given the recent advancements in SQLite's JSON capabilities.