4 ms·

They “open sourced” it enough (via the whitepaper) that huggingface is trying to reproduce their training now.

• dartos 2y ago
  How is it open source at all with no source? Paxos isn’t open source just because you can read the paxos paper.
  • famouswaffles 2y ago
    They released the weights
  • ComputerGuru 2y ago
    As an actual FOSS developer: they didn’t open source it. But I was merely adding the missing context using the (sorry) lingua Franca of AI.
  • nexus_six 2y ago
    For people who have the disciplinary background in neural networks and machine learning I imagine that replicating that paper into some type of framework would be straight forward right? Or am I mistaken?
    • jampekka 2y ago
      The model itself yes. The changes from previous architectures are often quite small code-wise. Quite often just adding/changing few lines in a torch model. Things like tweaking all the hyperparameters to make the training process actually work may be more tricky though.
  • ru552 2y ago
    With an LLM, the actual 0s and 1s of the model are fairly standard, common, and freely available to anyone that wants to use them. The "source code" for an LLM, is the process used to create the outcome, and to an extent, the data used to train with. DeepSeek released a highly detailed paper that describes the process used to create the outcome. People/Companies are actively trying to reproduce the work of DeepSeek to confirm the findings. It's more akin to scientific research where everyone is using the same molecules, but depending on the process you put the molecules through, you get a different outcome.
    • dartos 2y ago
      > With an LLM, the actual 0s and 1s of the model are fairly standard, common, and freely available to anyone that wants to use them How is that different than the 0s and 1s of a program? Assembly instructions are literally standard. What’s more, if said program uses something like Java, the byte code is even _more_ understandable. So much so that there is an ecosystem of Java decompilers. Binary files are not the “source” in question when talking about “open source”
      • fuzzbazz 2y ago
        There is no way to decompile an LLM's weights and obtain a somewhat meaningful, reproducible source, like with a program binary as you say. In fact, if we were to compare both in this way that would make a program binary more "open source".
        • dartos 2y ago
          Yes, that is my exact argument.
• jayd16 2y ago
  Publishing a white paper doesn't qualify as open source in any other context. Google Spanner has a nice white paper but you wouldn't consider it open source, for example.