9 ms·
TextSynth Server
- summarity 4y ago> The GPU version is commercial software. Please contact... Shame.
- copperx 4y agoShame as in "it's a shame" or as in "shame on them"?
- minxomat 4y agoThe former. Makes sense for their business model.
- dark-star 4y agoI'm pretty sure a dev skilled in ML and GPUs has no trouble modifying the MIT code (which runs on the CPU) to something that runs on GPUs...
- EMIRELADERO 4y agoThe CPU version is provided under binary format only.
- thetoon 4y agoYou're right : `The CPU version is released as binary code under the MIT license`. That's quite an unusual choice, not even sure how MIT would apply to that...
- Aissen 4y agoThe goal here is to allow you to redistribute while maintaining the copyright notice.
- capableweb 4y agoYou can release whatever you want under MIT :) It grants you the right to use the binary for commercial projects (or any type of projects you want), to modify the binary, distribute it yourself and more. You cannot hold Bellard/the license holder liable for anything related to it, and you must include the license and copyright if you distribute it. Seems pretty doable to me :)
- circuit10 4y agoI guess you could decompile it and redistribute that source if you were dedicated enough
- etaioinshrdlu 4y agoI think it's fine if a software legend (or anyone) wants to make some income.
- versteegen 4y agoThe GPU version of libNC is available as a free binary, and you can find MIT-licensed source code implementing (training and inference of) transformers using libNC at https://bellard.org/nncp https://bellard.org/nncp. (nncp was meant to be a submission to the Hutter Prize, which would have required open-sourcing libNC too, but it didn't qualify due to using AVX2 and too much RAM. At least the CPU version binary is MIT licensed since ts_server is.) I think it wouldn't be that big a project to support LLaMa starting from that code, although it is dense code. Edit: licensing
- sroecker 4y agoThat version of libnc_cuda.so doesn't seem to be compatible with ts_server though.
- rvz 4y ago"Shame. I cannot take the code / binary and run it as a SaaS efficiently." - HNers Good for him to commercialize it, and at least he is not pretending to be a non-profit accepting VC money.
- alexvoda 4y agoCountering SaaS-ifying by others can also be achieved through the AGPL or through the BSL (initially not open-source, reverts to open source after set period). I do believe one of the failures of GPL was not being AGPL from the start.
- Loic 4y agoFor me, the most interesting part is the statistics on all the models. These show that 8 bit quantization is basically as good as the full model and 4 bit is very close. This is the first time I see such table across a large number of models in one place.
- recuter 4y agoPretty much. Llama specific: https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa > According to GPTQ paper, As the size of the model increases, the difference in performance between FP16 and GPTQ decreases. https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-int-3-and https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i... https://docs.google.com/document/d/1wZ0g9rHI-6s7ctNlykuK4W5T-XijQXlZivY7_s-JR1E/edit https://docs.google.com/document/d/1wZ0g9rHI-6s7ctNlykuK4W5T... Expect to get away with a factor of 4-5 reduction in memory usage for a minimal loss of quality. :)
- canistel 4y agoThis man, Fabrice Bellard again... Frankly, I have not seen a more impressive portfolio of programming output.
- recuter 4y ago[flagged]
- ocimbote 4y ago[flagged]
- speedgoose 4y agoEveryone has an accent that tells where they are coming from. People who think they don’t have an accent simply don’t know that they have one.
- deleted 4y ago[deleted]
- shakow 4y agoI never understood why we triggered such hate on the Internet.
- recuter 4y ago
- JoachimS 4y agoThe Fabrice Bellard web page must be one of the most underselling ones on the entire web. So many amazing projects. Not a word that really emphasize the importance, coolness. Just a simple list with short factual descriptions.
- danwee 4y agoThese kind of people are like that. Does Linus Torvalds have a web page? No. He knows he doesn't need one. They are gods in the IT industry, and they know it.
- UncleEntity 4y agoHe has a blog because I recall reading about him reverse engineering a file format for his wife’s embroidery machine.
- ggerganov 4y agoVery inspiring stuff! I hope one day we get to see the magic behind libnc.
- hardwaresofton 4y agoI've been feeling FOMO (for lack of a better term) about recent AI & ML/GPT progression. It feels like ML/AI it might be the beginning of the end for a large class of things (if I wanted to be alarmist I'd say "everything") -- and the fact that Fabrice Bellard has jumped in and done the absolutely obvious rising-tide thing (building an API that abstracts the technologies) speaks volumes. Releasing something like this fits to Fabrice's pattern of work -- he built Qemu and that served as a similar enabling fabric for people to run virtual machines. QuickJS quietly powers some JS-on-another-platform functionality. Simon was right. The Stable Diffusion moment[0] is already here. It's going to accelerate. It was already moving at a speed that was hard to follow, and it's about to get even faster. There are too many world-changing things moving forward at the same time, and I'm only looking at such a small cut of the tech sphere. I don't know what to do with myself, I feel so thoroughly unprepared. [0]: https://simonwillison.net/2023/Mar/11/llama https://simonwillison.net/2023/Mar/11/llama
- lynx23 4y agoControl freak, by any chance? Is it really so hard to just "go with the flow"?
- arvinsim 4y agoHard to just "go with the flow" when your livelihood can be possibly threatened.
- yeahsure22 4y agoWell I’m sure that one guy will be able to change the course of history to suit his job situation. It’s going to happen so you had better go with the flow. Become water.
- DrewADesign 4y agoIn the US, people's jobs effectively justify their existence. Smugly saying mid or late career professionals can "become water" when their entire career category faces collapse is the most patronizing thing I've heard in a long time. That's a non-recoverable blow to many smart, capable people. There's a big difference between demanding we stop the wheels of progress to protect a few people and saying we owe the masses being crushed by them some harm reduction. Of course society on a whole will profit. But, the rust belt shows that the platitude about people deemed professionally unnecessary just "figuring it out" merely comforts the people turning those wheels.
- JacobiX 4y agoVery interesting as usual from Fabrice Bellard, but I'm a little bit disappointed this time, because libnc is a closed source DLL. Nevertheless it will be interesting to compare it to the amazing work of Georgi Gerganov: GGML Tensor Library. Both are heavily optimized, supports AVX intrinsics and are plain C/C++ implementation without dependencies.
- ggerganov 4y agoI expect LibNC will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap.
- hardwaresofton 4y agorefreshingly humble take -- thanks for your hard work. The work you've done and put out in the open is massive.
- liuliu 4y agoThe comparison table from the ts_server site looks awesome though. I wish we could generate one for llama.cpp, unfortunately too busy with other things at the moment.
- vidarh 4y agoChatGPT is pretty good at disassembling x86, and is able to give reasonable descriptions of what the code is doing (e.g try "disassemble the following bytes in hex and explain what they appear to be doing: [bytes from a binary in hex]") I'm curious how soon someone uses these models to effectively ruin the ability to use releasing binaries as an obfuscation method.
- DeathArrow 4y agoIf there were Oscars or Nobels for programming, Fabrice Bellard should have won one long ago!
- capableweb 4y agoI guess the ACM A. M. Turing Award is as close as we get, which AFAIK, they never won. But Bellard won countless of other awards, competitions and benchmarks, it is not like they are unrecognized for their work in the field.
- networked 4y ago> All is included in a single binary. Very few external dependencies (Python is not needed) so installation is easy on most Linux distributions. I have to disagree. The combination of being closed-source and dynamically linked makes a program a hassle to run on Linux. Even if it isn't at the moment of release, it soon becomes one. While ts_server is better than most, it already requires an old version of libjpeg-turbo not available in my distribution's repositories. I had to run it in a Rocky Linux container: docker run \ --rm \ --mount type=bind,source="$(pwd)",target=/app/ \ --publish 127.0.0.1:8080:8080 \ rockylinux:9 \ sh -c 'dnf install -y libjpeg libmicrohttpd && cd /app/ && ./ts_server ts_server.cfg' The solutions to this problem that I am aware of that do not involve releasing the source code are: 1) static linking; 2) containers; 3) shipping a Windows binary :-) ("Win32 is the only stable ABI on Linux" -- https://blog.hiler.eu/win32-the-only-stable-abi/ https://blog.hiler.eu/win32-the-only-stable-abi/).
- remram 4y agoThe ABI doesn't seem to be the problem, and Win32 does not include a libjpeg, so your Win32 approach would only work if it also bundles or statically links libjpeg.
- networked 4y ago> your Win32 approach would only work if it also bundles or statically links libjpeg. Of course. The stable ABI is to allow your bundled DLLs to keep functioning. (Check out https://news.ycombinator.com/item?id=32471624 https://news.ycombinator.com/item?id=32471624 for an extensive discussion of the link.)
- jefc1111 4y agoIn comments on this post, and elsewhere on other posts about AI, I see a lot of people referring to worries around the potential for lots of types of jobs to be heavily impacted by this technology. I feel like people are often referring to 'coding' when they express these worries. You know, actually writing code, having been given a spec to do so, and perhaps also participating in code review, writing tests, all the usual engineer stuff. My question is, amongst the HN crowd, what kinds of roles or areas do we think might be somewhat immune to this effect? The first thing that occurs to me are security, infrastructure & ops, networking. And of course the requirements gathering stage of software development. It is already the case that a lot of senior devs probably don't write much code and spend more time on communication between different stakeholders and overseeing whoever (or whatever) is writing the code. Anyone else been thinking about this? What tech roles might thrive in the face of AI.
- Accacin 4y agoIn my day job I work with TypeScript and React, and if I'm honest, I'm not worried at all. Why should I be? Automation is all around us, and has been for years. The way I personally see it, is that AI such as ChatGPT is another tool in our arsenal that we as developers will have to figure out how it fits into our workflow. I think long term it will help us write better code, and in general be more productive. For example, less time trying to find answers hidden deep in Stack Overflow as we'll be able to get that information directly from ChatGPT. I can completely see that some smaller places they might not require a developer and instead use ChatGPT to write code, but it still has to be verified and all the other processes around making that code "live", etc. If anything I'd be more worried if I were a copywriter, as I think it's an under appreciated skill and companies may think they can get away with ChatGPT and a quick glance over the copy. Either way, I'm positive and look for new ways to help me come a more productive and well-rounded programmer.
- jefc1111 4y agoI agree with this take, in the main. I think the level of FUD elsewhere is probably unwarranted. Though I fully subscribe to the idea that it is very disruptive tech. In the context of software engineering I am tending to see it as another layer of abstraction. Once upon a time there was perhaps not much above machine code / assembly, but now you can have quite a few layers providing abstractions over that ending up perhaps at Javascript, or maybe low-code tools. For me, AI sits somewhere vaguely in that category (though with a much higher level of sophistication).
- aww_dang 4y agoIs it necessary to pre-process models from hugging face before using them with libnc ?
- aww_dang 4y agoLooks like he previously provided a conversion script for his discontinued gpt2tc.
- deleted 4y ago[deleted]
- pierrec 4y ago"The CPU version is released as binary code under the MIT license" This gives off the surreal sci-fi vibe that the binary is the source. And who knows... true wizards work in mysterious ways.
- generalizations 4y agoIs it just me, or is the llama model not available to download there? Edit: nevermind, the models are all there, just some of the links aren't.
- a_subsystem 4y ago"...REST JSON API..." Please update. https://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven https://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypert...
- turmeric_root 4y agoI disagree with the linked post, most people use 'REST' to refer to JSON-over-HTTP now.
- a_subsystem 4y agoYou disagree with the guy who created the term REST?
- turmeric_root 4y agoYep.
- a_subsystem 4y ago'Most people' can call it what they want, but in this context, they're not referring to a RESTful API. Search for 'REST JSON API' and you may discover 'most people' may be a subset of 'some people', and not a majority of people.
- superkuh 4y agoAnyone know what format the models have to be in for use with textsynth? I looked at the gpt2 example binary (gpt2_117M.bin) and it seems like the "normal" params.json is embedded as a header for the binary and then some ascii string like "attn/c_attn/" and then the binary weights. I tried just using the Stanford Alpaca fine-tuned version of the llama 7B weights that work with llama.cpp with textsynth but it didn't like that (ggml-alpaca-7b-q4.bin: invalid file header). Having a textsynth HTTP API would save me a lot of hassle . I'm currently wrapping the stdin/out of a execution of a modified llama.cpp binary and that's extremely messy.
- ionflow 4y agoQuestion for you (sorry I don't have an answer for you): Where are you storing the models relative to the ts_server folder?