9 ms·

The dream could come back. In my opinion. Defeating legal threats: - Design server architecture so can be migrated from one country to a different one without

by faragon 11y ago

The dream could come back. In my opinion. Defeating legal threats: - Design server architecture so can be migrated from one country to a different one without censorship. Now is easy, with VPSs, you can move infrastructure in few hours, if you do automated deployment. Defeating cost threats ("run your own software like a 'megacorp' "): - Services running on cheap hardware handling 10-100x more users per server than current high language implementations (back to CGIs/Fast-CGIs on native code, instead of PHP/.NET/Java/NodeJS/Python/Perl). - Filesystem-less storage (start server, load filesystem to optimized in-process RAM DB), so following steps can be avoided: disk cache, filesystem tree search, multiple memory copies, etc. - Separate persistent and non-persistent data, so most operations don't need to hit the disk. - Abstraction over those low-level systems so people with high-level capacitation can build a massive-scale web application.

beagle3 11y ago
Two points I disagree with: > - Services running on cheap hardware handling 10-100x more users per server than current high language implementations (back to CGIs/Fast-CGIs on native code, instead of PHP/.NET/Java/NodeJS/Python/Perl). CGIs are slow even with native code, in the vast majority of cases, and FastCGIs are fast enough with properly written Python/Perl. Further, properly written Java or .NET, and even NodeJS performs well enough these days that unless you have some extreme case (you don't), you shouldn't even think about native. The tradeoff in bugs and dev time is just not worth it. > - Abstraction over those low-level systems so people with high-level capacitation can build a massive-scale web application. With incredibly consistent and repeatable correlation, abstractions over those low level system cost a lot in performance, thus contradicting your earlier points.
- faragon 11y ago
  > are slow even with native code, in the vast majority of cases, and FastCGIs are fast enough with properly written Python/Perl. (...) Yes, CGIs are slow because of the process spawn. About FastCGIs, if you want 10-100x throughput increase with same hardware, you have to go to a lower level. That's the tradeoff. I.e. properly written in native vs properly written in high level language. > The tradeoff in bugs and dev time is just not worth it. The tradeoff in bugs and dev time can be solved/increased with a higher level abstraction, e.g. using high level libraries, so the developer could avoid risks. > With incredibly consistent and repeatable correlation, abstractions over those low level system cost a lot in performance, thus contradicting your earlier points. No, if the abstraction is in the same low level language, e.g. using high level libraries on lower-level languages. In the end, higher level languages solve memory management, string handling, rich library, etc. With good libraries, C/C++/Others can be safer and as simple as e.g. NodeJS.
  - beagle3 11y ago
    > With good libraries, C/C++/Others can be safer and as simple as e.g. NodeJS. C cannot - there is apparently no way that is both reasonably efficient and reasonable foolproof to write C code; if you have large scale examples written by more than one person[0], I would love to see them. I concur C++ should be able to, but I again request large scale examples, because I've never seen any. Have you? > if you want 10-100x throughput increase with same hardware, you have to go to a lower level. That's the tradeoff. I.e. properly written in native vs properly written in high level language. You have to define your terms. Is Lua high or low level? Well written Lua code that does not need millions of small objects seems to run to within x2 as slow as comparable C code when executed through LuaJIT2. I've seen well written Perl beat well written C in the past (the Perl was better written, but both were well written). > The tradeoff in bugs and dev time can be solved/increased with a higher level abstraction, e.g. using high level libraries, so the developer could avoid risks. With a language that so easily lets you shoot yourself in the foot like C or C++, this is not the case in practice, see, e.g., every C program ever written. Can you give the examples you're thinking of so we can discuss them? [0] djb is an army all by himself, and even djb had security bugs - although, IIRC, by shear luck not exploitable on x86 or ARM.
    - faragon 11y ago
      > C cannot - there is apparently no way that is both reasonably efficient and reasonable foolproof to write C code; if you have large scale examples written by more than one person[0], I would love to see them. Not easy. Sure. In that regard, I'm trying something in order to help in the case of C, but it is not intended specifically for the problem we're speaking (just strings and other data structures, not a "NodeJS for C"): https://github.com/faragon/libsrt https://github.com/faragon/libsrt > I concur C++ should be able to, but I again request large scale examples, because I've never seen any. Have you? Of course not. There were many years ago when servers were way slower than today. The problem is not solved for current web, obviously. "Solved", in the sense of having some framework for having low level stuff in the backend without doing a byzantine effort. The point is about how to break monopolies in the David vs Goliath sense. If you have the resources, go high level. > You have to define your terms. Is Lua high or low level? Well written Lua code that does not need millions of small objects seems to run to within x2 as slow as comparable C code when executed through LuaJIT2. I've seen well written Perl beat well written C in the past (the Perl was better written, but both were well written). Please note that I was speaking about extreme optimization, even avoiding filesystem I/O, and reducing memory copies to the minimum. So it is not about comparing C vs Perl doing disk I/O, but in pure calculous. High level languages, even using JIT, have an important data structure size overhead and also allocation issues (when handling 10^5-10^6 requests per second on a single thread), so even with properly optimized code you can have poor performance in comparison to low level languages with optimized data structures (e.g. libraries using low level data structures using indexes instead pointers, multiplexing buffers, allowing memory aliasing for allowing conversions in the same buffer, etc.). > With a language that so easily lets you shoot yourself in the foot like C or C++, this is not the case in practice, see, e.g., every C program ever written. Can you give the examples you're thinking of so we can discuss them? Write backends in C/C++ is nonsense, unless you want to handle 10-100x requests using same hardware. If your service allows to cover the expense, go high level. Of course :-) P.S. some low level backend stuff: Kore: https://news.ycombinator.com/item?id=9558196 https://news.ycombinator.com/item?id=9558196 Rwasa: https://news.ycombinator.com/item?id=9948749 https://news.ycombinator.com/item?id=9948749 Redis: https://github.com/antirez/redis https://github.com/antirez/redis (e.g. changing it in order to call the DB operations directly from same-process, without doing connections)
nl 11y ago
Attempting sorely technical fixes to political problems is a delaying tactic, nothing more. Imagine a law like this: "You must be able to positively identify any person creating content on a service you control". Nothing you have suggested does anything at all to address that in anyway other than to try to make it harder to enforce. That method has been tried for 20 years, and is quickly becoming harder and harder. It's already extremely hard to buy a VPS with even pseudo-anonymity - most require credit cards, which means a law enforcement agency can retrieve your details trivially.
- anonbanker 11y ago
  i know of two iclandic outfits that accept bitcoin and anonymous registration. Nobody can trace my VPS to me.
  - mirimir 11y ago
    There are a few that accept Bitcoin.
  - nl 11y ago
    If you think Bitcoin keeps you anonymous in an age of ubiquitous network monitoring and growing work on Blockchain analysis then you have missed the point completely.
    - anonbanker 11y ago
      someone's never heard of a bitcoin laundry and mail2tor.
      - nl 11y ago
        Don't you see? This is exactly the point I'm making. I agree 100% that there will continue to be ways for extremely careful people to do this kind of thing. But the counter-measures and counter-counter-measures continue to ratchet up and at each step it more and more people are left behind. That's why I said Attempting sorely technical fixes to political problems is a delaying tactic, nothing more.
mercurial 11y ago
> Defeating cost threats ("run your own software like a 'megacorp' "): If you mean "run your own email server and decentralized social service node", you don't need high performance (note that you are still reliant on a third party to host your stuff anyway...). However, email in particular requires careful setup and maintenance, it's definitely more a problem of convenience than anything. Also, most people are not motivated enough to do that, even in the tech community, and existing social networks/IM systems have a huge network effect at that point.
- faragon 11y ago
  No. I mean services for world-wide audience (currently you need lots of datacenters), not for a local thing. If you don't need high performance, go higher level, of course.

by faragon 11y ago

beagle3 11y ago

Two points I disagree with: > - Services running on cheap hardware handling 10-100x more users per server than current high language implementations (back to CGIs/Fast-CGIs on native code, instead of PHP/.NET/Java/NodeJS/Python/Perl). CGIs are slow even with native code, in the vast majority of cases, and FastCGIs are fast enough with properly written Python/Perl. Further, properly written Java or .NET, and even NodeJS performs well enough these days that unless you have some extreme case (you don't), you shouldn't even think about native. The tradeoff in bugs and dev time is just not worth it. > - Abstraction over those low-level systems so people with high-level capacitation can build a massive-scale web application. With incredibly consistent and repeatable correlation, abstractions over those low level system cost a lot in performance, thus contradicting your earlier points.

faragon 11y ago
> are slow even with native code, in the vast majority of cases, and FastCGIs are fast enough with properly written Python/Perl. (...) Yes, CGIs are slow because of the process spawn. About FastCGIs, if you want 10-100x throughput increase with same hardware, you have to go to a lower level. That's the tradeoff. I.e. properly written in native vs properly written in high level language. > The tradeoff in bugs and dev time is just not worth it. The tradeoff in bugs and dev time can be solved/increased with a higher level abstraction, e.g. using high level libraries, so the developer could avoid risks. > With incredibly consistent and repeatable correlation, abstractions over those low level system cost a lot in performance, thus contradicting your earlier points. No, if the abstraction is in the same low level language, e.g. using high level libraries on lower-level languages. In the end, higher level languages solve memory management, string handling, rich library, etc. With good libraries, C/C++/Others can be safer and as simple as e.g. NodeJS.
- beagle3 11y ago
  > With good libraries, C/C++/Others can be safer and as simple as e.g. NodeJS. C cannot - there is apparently no way that is both reasonably efficient and reasonable foolproof to write C code; if you have large scale examples written by more than one person[0], I would love to see them. I concur C++ should be able to, but I again request large scale examples, because I've never seen any. Have you? > if you want 10-100x throughput increase with same hardware, you have to go to a lower level. That's the tradeoff. I.e. properly written in native vs properly written in high level language. You have to define your terms. Is Lua high or low level? Well written Lua code that does not need millions of small objects seems to run to within x2 as slow as comparable C code when executed through LuaJIT2. I've seen well written Perl beat well written C in the past (the Perl was better written, but both were well written). > The tradeoff in bugs and dev time can be solved/increased with a higher level abstraction, e.g. using high level libraries, so the developer could avoid risks. With a language that so easily lets you shoot yourself in the foot like C or C++, this is not the case in practice, see, e.g., every C program ever written. Can you give the examples you're thinking of so we can discuss them? [0] djb is an army all by himself, and even djb had security bugs - although, IIRC, by shear luck not exploitable on x86 or ARM.
  - faragon 11y ago
    > C cannot - there is apparently no way that is both reasonably efficient and reasonable foolproof to write C code; if you have large scale examples written by more than one person[0], I would love to see them. Not easy. Sure. In that regard, I'm trying something in order to help in the case of C, but it is not intended specifically for the problem we're speaking (just strings and other data structures, not a "NodeJS for C"): https://github.com/faragon/libsrt https://github.com/faragon/libsrt > I concur C++ should be able to, but I again request large scale examples, because I've never seen any. Have you? Of course not. There were many years ago when servers were way slower than today. The problem is not solved for current web, obviously. "Solved", in the sense of having some framework for having low level stuff in the backend without doing a byzantine effort. The point is about how to break monopolies in the David vs Goliath sense. If you have the resources, go high level. > You have to define your terms. Is Lua high or low level? Well written Lua code that does not need millions of small objects seems to run to within x2 as slow as comparable C code when executed through LuaJIT2. I've seen well written Perl beat well written C in the past (the Perl was better written, but both were well written). Please note that I was speaking about extreme optimization, even avoiding filesystem I/O, and reducing memory copies to the minimum. So it is not about comparing C vs Perl doing disk I/O, but in pure calculous. High level languages, even using JIT, have an important data structure size overhead and also allocation issues (when handling 10^5-10^6 requests per second on a single thread), so even with properly optimized code you can have poor performance in comparison to low level languages with optimized data structures (e.g. libraries using low level data structures using indexes instead pointers, multiplexing buffers, allowing memory aliasing for allowing conversions in the same buffer, etc.). > With a language that so easily lets you shoot yourself in the foot like C or C++, this is not the case in practice, see, e.g., every C program ever written. Can you give the examples you're thinking of so we can discuss them? Write backends in C/C++ is nonsense, unless you want to handle 10-100x requests using same hardware. If your service allows to cover the expense, go high level. Of course :-) P.S. some low level backend stuff: Kore: https://news.ycombinator.com/item?id=9558196 https://news.ycombinator.com/item?id=9558196 Rwasa: https://news.ycombinator.com/item?id=9948749 https://news.ycombinator.com/item?id=9948749 Redis: https://github.com/antirez/redis https://github.com/antirez/redis (e.g. changing it in order to call the DB operations directly from same-process, without doing connections)

nl 11y ago

Attempting sorely technical fixes to political problems is a delaying tactic, nothing more. Imagine a law like this: "You must be able to positively identify any person creating content on a service you control". Nothing you have suggested does anything at all to address that in anyway other than to try to make it harder to enforce. That method has been tried for 20 years, and is quickly becoming harder and harder. It's already extremely hard to buy a VPS with even pseudo-anonymity - most require credit cards, which means a law enforcement agency can retrieve your details trivially.

anonbanker 11y ago
i know of two iclandic outfits that accept bitcoin and anonymous registration. Nobody can trace my VPS to me.
- mirimir 11y ago
  There are a few that accept Bitcoin.
- nl 11y ago
  If you think Bitcoin keeps you anonymous in an age of ubiquitous network monitoring and growing work on Blockchain analysis then you have missed the point completely.
  - anonbanker 11y ago
    someone's never heard of a bitcoin laundry and mail2tor.
    - nl 11y ago
      Don't you see? This is exactly the point I'm making. I agree 100% that there will continue to be ways for extremely careful people to do this kind of thing. But the counter-measures and counter-counter-measures continue to ratchet up and at each step it more and more people are left behind. That's why I said Attempting sorely technical fixes to political problems is a delaying tactic, nothing more.

mercurial 11y ago

> Defeating cost threats ("run your own software like a 'megacorp' "): If you mean "run your own email server and decentralized social service node", you don't need high performance (note that you are still reliant on a third party to host your stuff anyway...). However, email in particular requires careful setup and maintenance, it's definitely more a problem of convenience than anything. Also, most people are not motivated enough to do that, even in the tech community, and existing social networks/IM systems have a huge network effect at that point.

faragon 11y ago
No. I mean services for world-wide audience (currently you need lots of datacenters), not for a local thing. If you don't need high performance, go higher level, of course.