10 ms·

Weights are data, not a type of program. A computer program is a set of instructions that may be executed. Weights are values that may be loaded by a program,

Weights are data, not a type of program. A computer program is a set of instructions that may be executed. Weights are values that may be loaded by a program, but are not a program in and of themselves.

jstanley 3y ago
It's a very difficult distinction to make. Would you consider a Python program to be data rather than program just because it is text input to the python interpreter instead of machine code for the CPU?
- slowmovintarget 3y ago
  It is not at all a difficult distinction. Weights are literally numbers computed as output. They are not instructions. The semantics of those numbers even when emplaced (loaded) in an artificial neural net is such that they do not execute. They are not instructions. LLM engines and diffusers perform searches where the weights are used to calculate additional output. Is source code, like Python text, data? Yes. All code is data. But not all data are source code. If I gave you a web request log, you would not assert it is a program. If I gave you a CSV file with time-series values from a sensor, you would not assert it is a program. If I hand you a database of contact information, you would not assert it is a program. Weight files are the equivalent of CSV files. They are are a dump of parameter values computed from training. They are not a program. The definition of computer program is well worn. So is the definition of source code, and the definition of parameters. Weights are parameters.
  - jstanley 3y ago
    The difference between code and data only exists in our minds. There is no distinction. Both code and data make the computer do things (and, yes, both code and data only make the computer do things if other conditions are permitting, for example if executed with the right interpreter, or loaded with the right type of viewer). Anything that can be expressed as code can be expressed as data, and vice versa.
  - xigoi 3y ago
    If a program has to consist of instructions, then source code written in a declarative language is not a program.
  - pravus 3y ago
    > Weights are literally numbers computed as output. They are not instructions. They are instructions if you consider the LLM system itself to be a kind of weird, indirect virtual machine. Each number can be mapped to a set of instructions that are executed. Even your CPU uses numbers (machine codes) to execute. Join me in saying: ...code is data is code is data is code is data...
graypegg 3y ago
They’re not data though, they’re coefficients. They are the only thing that significantly differentiates one model from another. If I told you the economy can be accurately modelled by GDP(x) = Ax + B But I don’t define A And B for you because it’s proprietary, you haven’t learned anything other than what you can glean from the structure of the model itself (it’s linear, there’s only a single input etc) If most of these models are similarly structured, I’d say the weights are the program.
- slowmovintarget 3y ago
  The nature of the data as proprietary or not, important or not, is not relevant. Parameters, or actual arguments, are values; data. Not instructions. Valuable data is still data. It's significance doesn't magically turn it into source code.
golemotron 3y ago
No, declarative programs exist. They are not instructions. There is no real line between code and data. This is an observation that runs all the way from Turing Machines in computability theory to the Von Neumann architecture and homoiconicity in Lisp. What we call 'data' is just code that needs a cleverer interpreter.
- Izkata 3y ago
  Less into theory and more into "wait wtf": Some of the older projects I've worked on were written by people who loved database-driven stuff, to the point they did things like put perl code into one table column (with sentinel values you had to find/replace before `eval`ing the code) and sql into another table that retrieved values for those find/replaces, both retrieved and executed by some really generic code. Code or data: Well... both.
killjoywashere 3y ago
But not "raw" data. They are derived from other data and a program. If this was a collaboration where one collaborator did the processing and one sourced the data, they would likely both claim some amount of ownership of the trained weights. At a minimum, it would be an active area of negotiation that the attorneys would take notice of. Source: have negotiated these agreements.
- slowmovintarget 3y ago
  A curated data set is still a data set. I imagine it is not settled law, but there's a clear argument to be made that regardless of the difficulty in curating the data set, it's still a data set. Can it be licensed and sold. Yes, surely. Is it proper to pretend an open source license is sufficient protection, probably not.
programmarchy 3y ago
This is a distinction without a difference. Code is data and data is code.
- mrguyorama 3y ago
  Maybe they've only worked with machines using a Harvard Architecture
- slowmovintarget 3y ago
  All source code is data. Not all data is source code. Data may be encoded, but that doesn't make it source code either.
earleybird 3y ago
Weights are data in the same way that instruction codes in memory is data.
- slowmovintarget 3y ago
  Values for the variables do not the function make.
  - daniel-cussen 3y ago
    That's the least of it. In Lisp the distinction between code n data is blurred all the time. In F18 assembly i frequently have "double entendres" which are used as code or as literals depending on the entry point. I think at least once there was code and data in the same entry point. Assembly n Lisp are both homoiconic, after all. N verb at the end of the sentence, are you transliterating German, or a two-foot green Jedi master full of wisdom?
rockinghigh 3y ago
When people talk about weights, they talk about a network of weights that takes an input and computes an output. There is really not much difference between a saved model and a program.

jstanley 3y ago

It's a very difficult distinction to make. Would you consider a Python program to be data rather than program just because it is text input to the python interpreter instead of machine code for the CPU?

slowmovintarget 3y ago
It is not at all a difficult distinction. Weights are literally numbers computed as output. They are not instructions. The semantics of those numbers even when emplaced (loaded) in an artificial neural net is such that they do not execute. They are not instructions. LLM engines and diffusers perform searches where the weights are used to calculate additional output. Is source code, like Python text, data? Yes. All code is data. But not all data are source code. If I gave you a web request log, you would not assert it is a program. If I gave you a CSV file with time-series values from a sensor, you would not assert it is a program. If I hand you a database of contact information, you would not assert it is a program. Weight files are the equivalent of CSV files. They are are a dump of parameter values computed from training. They are not a program. The definition of computer program is well worn. So is the definition of source code, and the definition of parameters. Weights are parameters.
- jstanley 3y ago
  The difference between code and data only exists in our minds. There is no distinction. Both code and data make the computer do things (and, yes, both code and data only make the computer do things if other conditions are permitting, for example if executed with the right interpreter, or loaded with the right type of viewer). Anything that can be expressed as code can be expressed as data, and vice versa.
- xigoi 3y ago
  If a program has to consist of instructions, then source code written in a declarative language is not a program.
- pravus 3y ago
  > Weights are literally numbers computed as output. They are not instructions. They are instructions if you consider the LLM system itself to be a kind of weird, indirect virtual machine. Each number can be mapped to a set of instructions that are executed. Even your CPU uses numbers (machine codes) to execute. Join me in saying: ...code is data is code is data is code is data...

graypegg 3y ago

They’re not data though, they’re coefficients. They are the only thing that significantly differentiates one model from another. If I told you the economy can be accurately modelled by GDP(x) = Ax + B But I don’t define A And B for you because it’s proprietary, you haven’t learned anything other than what you can glean from the structure of the model itself (it’s linear, there’s only a single input etc) If most of these models are similarly structured, I’d say the weights are the program.

slowmovintarget 3y ago
The nature of the data as proprietary or not, important or not, is not relevant. Parameters, or actual arguments, are values; data. Not instructions. Valuable data is still data. It's significance doesn't magically turn it into source code.

golemotron 3y ago

No, declarative programs exist. They are not instructions. There is no real line between code and data. This is an observation that runs all the way from Turing Machines in computability theory to the Von Neumann architecture and homoiconicity in Lisp. What we call 'data' is just code that needs a cleverer interpreter.

Izkata 3y ago
Less into theory and more into "wait wtf": Some of the older projects I've worked on were written by people who loved database-driven stuff, to the point they did things like put perl code into one table column (with sentinel values you had to find/replace before `eval`ing the code) and sql into another table that retrieved values for those find/replaces, both retrieved and executed by some really generic code. Code or data: Well... both.

killjoywashere 3y ago

But not "raw" data. They are derived from other data and a program. If this was a collaboration where one collaborator did the processing and one sourced the data, they would likely both claim some amount of ownership of the trained weights. At a minimum, it would be an active area of negotiation that the attorneys would take notice of. Source: have negotiated these agreements.

slowmovintarget 3y ago
A curated data set is still a data set. I imagine it is not settled law, but there's a clear argument to be made that regardless of the difficulty in curating the data set, it's still a data set. Can it be licensed and sold. Yes, surely. Is it proper to pretend an open source license is sufficient protection, probably not.

programmarchy 3y ago

This is a distinction without a difference. Code is data and data is code.

mrguyorama 3y ago
Maybe they've only worked with machines using a Harvard Architecture
slowmovintarget 3y ago
All source code is data. Not all data is source code. Data may be encoded, but that doesn't make it source code either.

earleybird 3y ago

Weights are data in the same way that instruction codes in memory is data.

slowmovintarget 3y ago
Values for the variables do not the function make.
- daniel-cussen 3y ago
  That's the least of it. In Lisp the distinction between code n data is blurred all the time. In F18 assembly i frequently have "double entendres" which are used as code or as literals depending on the entry point. I think at least once there was code and data in the same entry point. Assembly n Lisp are both homoiconic, after all. N verb at the end of the sentence, are you transliterating German, or a two-foot green Jedi master full of wisdom?

rockinghigh 3y ago

When people talk about weights, they talk about a network of weights that takes an input and computes an output. There is really not much difference between a saved model and a program.