4 ms·

The software patent situation is getting more and more ridiculous by the day. This patent is so general, it describes basically any data-mining software (that m

The software patent situation is getting more and more ridiculous by the day. This patent is so general, it describes basically any data-mining software (that may or may not be distributed over multiple computers.) I think that all patents should have to go through the peer review process, unless there is some kind of extra-ordinary reason that it can't be made public: http://www.peertopatent.org/ http://www.peertopatent.org/

cperciva 17y ago
This patent is so general, it describes basically any data-mining software (that may or may not be distributed over multiple computers.) Bullshit. Here are the two independent claims: 1. A system for large-scale processing of data, comprising: a plurality of processes executing on a plurality of interconnected processors; the plurality of processes including a master process, for coordinating a data processing job for processing a set of input data, and worker processes; the master process, in response to a request to perform the data processing job, assigning input data blocks of the set of input data to respective ones of the worker processes; each of a first plurality of the worker processes including an application-independent map module for retrieving a respective input data block assigned to the worker process by the master process and applying an application-specific map operation to the respective input data block to produce intermediate data values, wherein at least a subset of the intermediate data values each comprises a key/value pair, and wherein at least two of the first plurality of the worker processes operate simultaneously so as to perform the application-specific map operation in parallel on distinct, respective input data blocks; a partition operator for processing the produced intermediate data values to produce a plurality of intermediate data sets, wherein each respective intermediate data set includes all key/value pairs for a distinct set of respective keys, and wherein at least one of the respective intermediate data sets includes respective ones of the key/value pairs produced by a plurality of the first plurality of the worker processes; and each of a second plurality of the worker processes including an application-independent reduce module for retrieving data, the retrieved data comprising at least a subset of the key/value pairs from a respective intermediate data set of the plurality of intermediate data sets and applying an application-specific reduce operation to the retrieved data to produce final output data corresponding to the distinct set of respective keys in the respective intermediate data set of the plurality of intermediate data sets, and wherein at least two of the second plurality of the worker processes operate simultaneously so as to perform the application-specific reduce operation in parallel on multiple respective subsets of the produced intermediate data values. 9. A method of performing a large-scale data processing job, comprising: executing a plurality of processes on a plurality of interconnected processors, the plurality of processes including a master process for coordinating the large-scale data processing job for processing a set of input data, and worker processes; in the master process, in response to a request to perform the large-scale data processing job, assigning input data blocks of the set of input data to respective ones of the worker processes; in each of a first plurality of the worker processes, executing an application-independent map module to retrieve a respective input data block assigned to the worker process by the master process and to apply an application-specific map operation to the respective input data block to produce intermediate data values, wherein at least a subset of the intermediate data values each comprises a key/value pair, and wherein at least two of the first plurality of the worker processes operate simultaneously so as to perform the application-specific map operation in parallel on distinct, respective input data blocks; using a partition operator to process the produced intermediate data values to produce a plurality of intermediate data sets, wherein each respective intermediate data set includes all key/value pairs for a distinct set of respective keys, and wherein at least one of the respective intermediate data sets includes respective ones of the key/value pairs produced by a plurality of the first plurality of the worker processes; and in each of a second plurality of the worker processes, executing an application-independent reduce module to retrieve data, the retrieved data comprising at least a subset of the key/value pairs from a respective intermediate data set of the plurality of intermediate data sets and applying an application-specific reduce operation to the retrieved data to produce final output data corresponding to the distinct set of respective keys in the respective intermediate data set of the plurality of intermediate data sets, and wherein at least two of the second plurality of the worker processes operate simultaneously so as to perform the application-specific reduce operation in parallel on multiple respective subsets of the produced intermediate data values. This very clearly only covers computations distributed over multiple systems (a plurality of processes executing on a plurality of interconnected processors) and is narrow enough that there are lots of parallel systems which don't fall under those claims.
- tumult 17y ago
  It looks like doing a foldr with key pairs and then a foldl violates that patent. In parallel Haskell, it can run on multiple processors, with the assistance of the RTS ('master process'). It's bullshit.
- ajross 17y ago
  No offense, but shouldn't the standard be a little higher than "there are lots of parallel systems which don't fall under those claims"? It looks quite shockingly broad to me.
  - cperciva 17y ago
    The standard for patenting is higher than that. I was responding to the claim that the patent "describes basically any data-mining software", which is simply untrue.
- profquail 17y ago
  "a plurality of processes executing on a plurality of interconnected processors" could describe both interconnected machines, and multicore processors. For example, does this patent cover doing a data reduction on a GPU? It sounds like it would -- for example, the Thrust library: http://code.google.com/p/thrust/ http://code.google.com/p/thrust/ does basically those two "claims", just on a GPU (which falls under your definition). In any case, my complaint is more about the wording than the actual technology that the patent covers. Patents are granted for such generic technology these days, that it's going to scare away people from trying to create new things (lest it already be covered under a generic patent that really doesn't relate to the new product.)
tghw 17y ago
They are patenting MapReduce. Seems pretty specific to me.
- loup-vaillant 17y ago
  The Idea behind Map Reduce itself is extremely general (and not brand new): map :: (a -> b) -> [a] -> b fold :: (a -> a -> a) -> a -> [a] -> a -- the first argument must be associative if you want a parallel fold. mapReduce :: (a -> b) -> (b -> b -> b) -> b -> [a] -> b mapReduce f g e l = fold g e (map f l) -- conceptually. You have to write an actually parallel version. If the patent isn't more specific than this (by being limited to MapReduce, for instance), I bet it can be overruled by prior art. Now, I admit I don't have the courage to check.
  - neilc 17y ago
    A cursory examination of the patent would tell you that it says very little about map and reduce functions as such -- the emphasis is on the system for executing such functions in parallel for large-scale data transformation.
    - ramchip 17y ago
      the emphasis is on the system for executing such functions in parallel for large-scale data transformation As someone mentioned: the Glasgow Haskell Compiler and its RTS is such a system...

cperciva 17y ago

This patent is so general, it describes basically any data-mining software (that may or may not be distributed over multiple computers.) Bullshit. Here are the two independent claims: 1. A system for large-scale processing of data, comprising: a plurality of processes executing on a plurality of interconnected processors; the plurality of processes including a master process, for coordinating a data processing job for processing a set of input data, and worker processes; the master process, in response to a request to perform the data processing job, assigning input data blocks of the set of input data to respective ones of the worker processes; each of a first plurality of the worker processes including an application-independent map module for retrieving a respective input data block assigned to the worker process by the master process and applying an application-specific map operation to the respective input data block to produce intermediate data values, wherein at least a subset of the intermediate data values each comprises a key/value pair, and wherein at least two of the first plurality of the worker processes operate simultaneously so as to perform the application-specific map operation in parallel on distinct, respective input data blocks; a partition operator for processing the produced intermediate data values to produce a plurality of intermediate data sets, wherein each respective intermediate data set includes all key/value pairs for a distinct set of respective keys, and wherein at least one of the respective intermediate data sets includes respective ones of the key/value pairs produced by a plurality of the first plurality of the worker processes; and each of a second plurality of the worker processes including an application-independent reduce module for retrieving data, the retrieved data comprising at least a subset of the key/value pairs from a respective intermediate data set of the plurality of intermediate data sets and applying an application-specific reduce operation to the retrieved data to produce final output data corresponding to the distinct set of respective keys in the respective intermediate data set of the plurality of intermediate data sets, and wherein at least two of the second plurality of the worker processes operate simultaneously so as to perform the application-specific reduce operation in parallel on multiple respective subsets of the produced intermediate data values. 9. A method of performing a large-scale data processing job, comprising: executing a plurality of processes on a plurality of interconnected processors, the plurality of processes including a master process for coordinating the large-scale data processing job for processing a set of input data, and worker processes; in the master process, in response to a request to perform the large-scale data processing job, assigning input data blocks of the set of input data to respective ones of the worker processes; in each of a first plurality of the worker processes, executing an application-independent map module to retrieve a respective input data block assigned to the worker process by the master process and to apply an application-specific map operation to the respective input data block to produce intermediate data values, wherein at least a subset of the intermediate data values each comprises a key/value pair, and wherein at least two of the first plurality of the worker processes operate simultaneously so as to perform the application-specific map operation in parallel on distinct, respective input data blocks; using a partition operator to process the produced intermediate data values to produce a plurality of intermediate data sets, wherein each respective intermediate data set includes all key/value pairs for a distinct set of respective keys, and wherein at least one of the respective intermediate data sets includes respective ones of the key/value pairs produced by a plurality of the first plurality of the worker processes; and in each of a second plurality of the worker processes, executing an application-independent reduce module to retrieve data, the retrieved data comprising at least a subset of the key/value pairs from a respective intermediate data set of the plurality of intermediate data sets and applying an application-specific reduce operation to the retrieved data to produce final output data corresponding to the distinct set of respective keys in the respective intermediate data set of the plurality of intermediate data sets, and wherein at least two of the second plurality of the worker processes operate simultaneously so as to perform the application-specific reduce operation in parallel on multiple respective subsets of the produced intermediate data values. This very clearly only covers computations distributed over multiple systems (a plurality of processes executing on a plurality of interconnected processors) and is narrow enough that there are lots of parallel systems which don't fall under those claims.

tumult 17y ago
It looks like doing a foldr with key pairs and then a foldl violates that patent. In parallel Haskell, it can run on multiple processors, with the assistance of the RTS ('master process'). It's bullshit.
ajross 17y ago
No offense, but shouldn't the standard be a little higher than "there are lots of parallel systems which don't fall under those claims"? It looks quite shockingly broad to me.
- cperciva 17y ago
  The standard for patenting is higher than that. I was responding to the claim that the patent "describes basically any data-mining software", which is simply untrue.
profquail 17y ago
"a plurality of processes executing on a plurality of interconnected processors" could describe both interconnected machines, and multicore processors. For example, does this patent cover doing a data reduction on a GPU? It sounds like it would -- for example, the Thrust library: http://code.google.com/p/thrust/ http://code.google.com/p/thrust/ does basically those two "claims", just on a GPU (which falls under your definition). In any case, my complaint is more about the wording than the actual technology that the patent covers. Patents are granted for such generic technology these days, that it's going to scare away people from trying to create new things (lest it already be covered under a generic patent that really doesn't relate to the new product.)

tghw 17y ago

They are patenting MapReduce. Seems pretty specific to me.

loup-vaillant 17y ago
The Idea behind Map Reduce itself is extremely general (and not brand new): map :: (a -> b) -> [a] -> b fold :: (a -> a -> a) -> a -> [a] -> a -- the first argument must be associative if you want a parallel fold. mapReduce :: (a -> b) -> (b -> b -> b) -> b -> [a] -> b mapReduce f g e l = fold g e (map f l) -- conceptually. You have to write an actually parallel version. If the patent isn't more specific than this (by being limited to MapReduce, for instance), I bet it can be overruled by prior art. Now, I admit I don't have the courage to check.
- neilc 17y ago
  A cursory examination of the patent would tell you that it says very little about map and reduce functions as such -- the emphasis is on the system for executing such functions in parallel for large-scale data transformation.
  - ramchip 17y ago
    the emphasis is on the system for executing such functions in parallel for large-scale data transformation As someone mentioned: the Glasgow Haskell Compiler and its RTS is such a system...