5 ms·

is it me or is this going to result in massively long data files that you KEEP having to re-downloading in their entirety every time you want to get an update f

is it me or is this going to result in massively long data files that you KEEP having to re-downloading in their entirety every time you want to get an update from someone you follow?

akkartik 11y ago
https://en.wikipedia.org/wiki/HTTP_ETag https://en.wikipedia.org/wiki/HTTP_ETag should help with that.
- hk__2 11y ago
  It can help to avoid re-downloading a timeline that didn’t change but you’ll still have to re-download the whole file for each new status.
  - deleted 11y ago
    [deleted]
  - detaro 11y ago
    Make the ETAG the size, send a range request for only the bytes you haven't seen already?
    - buckket 11y ago
      This is nice as long as nobody changes old status updates. Thought about using If-Modified-Since to at least reduce the load when nothing has changed at all. But still: It will definitely take some time till a "twtxt timeline" causes more traffic than browsing the current Twitter homepage. :3
pythousiast 11y ago
Yup that's an issue. Their should be more control on how the data is stored. I like the concept tho.
- sleepychu 11y ago
  Can't you just use an HTTP range request?
  - buckket 11y ago
    Yes, should be possible. But then again you would have to specify an ordering, i.e. are newer status updates at the top or at the bottom.
klapinat0r 11y ago
This could be solved by archiving/rotating old entries, and have the webserver return the n latest. For instance, you could let nginx do this: location /masukomi.txt { default_type 'text/plain'; content_by_lua 'local tail = io.popen("tail -n 20 ".."/chroot/twtxt/masukomi.txt", "r") for line in tail:lines() do print(c, line) end '; }
- toyg 11y ago
  The problem is that if you didn't fetch the file at the right time, you will lose some history after rotation happens. At the very least, it needs a standard way to deal with that (like a mandated date-based URL format?), which has to be specified in the format. Also, no DM and no @-replies. For those, you need a registry of some sort, but decentralized registries enable name-squatting. Before you know it, you have reinvented the WWW, except limited to 140 chars.
  - klapinat0r 11y ago
    > if you didn't fetch the file at the right time, you will lose some history after rotation Very true, and that's a trade-off, not unlike what Twitter does today. In my pseudo-code I didn't make the -n part of the url (it easily could be, but that'd add changes to the client). So I agree with your criticism, but a "since" lookup would be a fairly easy extension of the client (i.e. part of query, so unsupported frontends still work), as the data file already contains date. > Before you know it, you have reinvented the WWW Yes. :)
- amelius 11y ago
  No need for all this complication. Simply use IPFS[1] instead as the underlying protocol. [1] https://ipfs.io/ https://ipfs.io/

is it me or is this going to result in massively long data files that you KEEP having to re-downloading in their entirety every time you want to get an update from someone you follow?

akkartik 11y ago

https://en.wikipedia.org/wiki/HTTP_ETag https://en.wikipedia.org/wiki/HTTP_ETag should help with that.

hk__2 11y ago
It can help to avoid re-downloading a timeline that didn’t change but you’ll still have to re-download the whole file for each new status.
- deleted 11y ago
  [deleted]
- detaro 11y ago
  Make the ETAG the size, send a range request for only the bytes you haven't seen already?
  - buckket 11y ago
    This is nice as long as nobody changes old status updates. Thought about using If-Modified-Since to at least reduce the load when nothing has changed at all. But still: It will definitely take some time till a "twtxt timeline" causes more traffic than browsing the current Twitter homepage. :3

pythousiast 11y ago

Yup that's an issue. Their should be more control on how the data is stored. I like the concept tho.

sleepychu 11y ago
Can't you just use an HTTP range request?
- buckket 11y ago
  Yes, should be possible. But then again you would have to specify an ordering, i.e. are newer status updates at the top or at the bottom.

klapinat0r 11y ago

This could be solved by archiving/rotating old entries, and have the webserver return the n latest. For instance, you could let nginx do this: location /masukomi.txt { default_type 'text/plain'; content_by_lua 'local tail = io.popen("tail -n 20 ".."/chroot/twtxt/masukomi.txt", "r") for line in tail:lines() do print(c, line) end '; }

toyg 11y ago
The problem is that if you didn't fetch the file at the right time, you will lose some history after rotation happens. At the very least, it needs a standard way to deal with that (like a mandated date-based URL format?), which has to be specified in the format. Also, no DM and no @-replies. For those, you need a registry of some sort, but decentralized registries enable name-squatting. Before you know it, you have reinvented the WWW, except limited to 140 chars.
- klapinat0r 11y ago
  > if you didn't fetch the file at the right time, you will lose some history after rotation Very true, and that's a trade-off, not unlike what Twitter does today. In my pseudo-code I didn't make the -n part of the url (it easily could be, but that'd add changes to the client). So I agree with your criticism, but a "since" lookup would be a fairly easy extension of the client (i.e. part of query, so unsupported frontends still work), as the data file already contains date. > Before you know it, you have reinvented the WWW Yes. :)
amelius 11y ago
No need for all this complication. Simply use IPFS[1] instead as the underlying protocol. [1] https://ipfs.io/ https://ipfs.io/