3 ms·

I tried creating 'Peanut Gallery' about a decade ago. The main issues then were trying to figure out the best interface that could work across browsers. I also

I tried creating 'Peanut Gallery' about a decade ago. The main issues then were trying to figure out the best interface that could work across browsers. I also didn't know how to scale the solution up to allow it to work on every web page for every users. Also, I anticipated that there would be lots of privacy concerns as you'd basically be collecting every web page a user went to.

kenshaw 12y ago
1. Scaling this would be a combination of selecting the right data store that would scale linearly on performance as well as on storage costs. There are many. 2. If you were concerned about privacy issues by exposing URLs, you would simply do the following: a) pass URLs only as hashes of the original text, b) have a policy of only storing URLs if they contained annotations This is actually fairly trivial to do right.
- jsprogrammer 12y ago
  1. What data store can scale linearly on performance and storage (with no bounds)? I'm not sure it exists (or is even possible). We decided on horizontal partitioning with hashed URLs/IPs as the key. 2. We also wanted to provide real-time stats for pages, sites, and hosts. This requires IDing the user and page in a reversible way. Even if it's not reversible, you could easily use a rainbow table to reverse, especially with the limited IPv4 namespace. However, it seems that such URL collection isn't too big of a concern among the general population. You also have to maintain plugins or extensions across all the major browsers, which is a pain. This was a project I was working on before completing a CS degree and/or getting exposure to some of the solutions that could be used. I'd be interested in exploring it again, but I have a few more interesting projects on the stack for now.
  - kenshaw 12y ago
    Cassandra when done correctly, will scale linearly[0]. As will most other Dynamo type data stores. Additionally, there are many other large databases (Hadoop, etc) that will scale linearly. While it may not be ideal for this use case, but a proper multi-master setup with many read slaves in either PostgreSQL or MySQL could be described as being able to scale linearly to some unspecified extremely large upper limit (see work done previously at Yahoo, Google, Amazon, et al). Much of this also depends on what you define as scaling linearly (are you looking for linear scaling for memory, reads, writes, storage, latency, costs, etc?). Real time stats for pages, sites, hosts for a shared annotation system across browsers doesn't mesh well with your initial condition of "privacy concerns." [0] http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
    - jsprogrammer 12y ago
      The Cassandra benchmarks were only for ~1 million writes/sec. I think you'd need to support around 1 billion writes/sec for this application. I think it's possible. The "privacy concerns" weren't necessarily my own, I just anticipated that others might have them and therefore not adopt the system. For me, the point was to share what sites you were visiting because you'd be annotating (well, chat/message-boarding) those sites with others. Additionally, I thought it would be cool to see who was where at a given time, what sites were popular, or which pages people found particularly interesting (interesting enough to visit anyway) at any given time.
      - kenshaw 12y ago
        You get diminishing returns, of course, on any kind of technology. You would just basically partition this data out to a 1000 different clusters then, each capable of supporting 1 million writes/second. That would be done through an algorithm along the lines of this: partition_id = well_distributed_hash(normalize_domain_name(<domain>)) really don't think this would be an issue, or even a concern, as 1 billion writes per second are "Google-levels" of writes. One would likely run out of financial resources supporting an expansion of 86 terrabytes / day (assuming 1kb minimum new data per write), far faster than you would ever max out the capabilities of the technology. It's more a finance/business exercise to determine if revenue inflows would actually meet, cover, or exceed revenue outflows on a project like this. You can always make the technology scale and meet the demand.

kenshaw 12y ago

1. Scaling this would be a combination of selecting the right data store that would scale linearly on performance as well as on storage costs. There are many. 2. If you were concerned about privacy issues by exposing URLs, you would simply do the following: a) pass URLs only as hashes of the original text, b) have a policy of only storing URLs if they contained annotations This is actually fairly trivial to do right.

jsprogrammer 12y ago
1. What data store can scale linearly on performance and storage (with no bounds)? I'm not sure it exists (or is even possible). We decided on horizontal partitioning with hashed URLs/IPs as the key. 2. We also wanted to provide real-time stats for pages, sites, and hosts. This requires IDing the user and page in a reversible way. Even if it's not reversible, you could easily use a rainbow table to reverse, especially with the limited IPv4 namespace. However, it seems that such URL collection isn't too big of a concern among the general population. You also have to maintain plugins or extensions across all the major browsers, which is a pain. This was a project I was working on before completing a CS degree and/or getting exposure to some of the solutions that could be used. I'd be interested in exploring it again, but I have a few more interesting projects on the stack for now.
- kenshaw 12y ago
  Cassandra when done correctly, will scale linearly[0]. As will most other Dynamo type data stores. Additionally, there are many other large databases (Hadoop, etc) that will scale linearly. While it may not be ideal for this use case, but a proper multi-master setup with many read slaves in either PostgreSQL or MySQL could be described as being able to scale linearly to some unspecified extremely large upper limit (see work done previously at Yahoo, Google, Amazon, et al). Much of this also depends on what you define as scaling linearly (are you looking for linear scaling for memory, reads, writes, storage, latency, costs, etc?). Real time stats for pages, sites, hosts for a shared annotation system across browsers doesn't mesh well with your initial condition of "privacy concerns." [0] http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
  - jsprogrammer 12y ago
    The Cassandra benchmarks were only for ~1 million writes/sec. I think you'd need to support around 1 billion writes/sec for this application. I think it's possible. The "privacy concerns" weren't necessarily my own, I just anticipated that others might have them and therefore not adopt the system. For me, the point was to share what sites you were visiting because you'd be annotating (well, chat/message-boarding) those sites with others. Additionally, I thought it would be cool to see who was where at a given time, what sites were popular, or which pages people found particularly interesting (interesting enough to visit anyway) at any given time.
    - kenshaw 12y ago
      You get diminishing returns, of course, on any kind of technology. You would just basically partition this data out to a 1000 different clusters then, each capable of supporting 1 million writes/second. That would be done through an algorithm along the lines of this: partition_id = well_distributed_hash(normalize_domain_name(<domain>)) really don't think this would be an issue, or even a concern, as 1 billion writes per second are "Google-levels" of writes. One would likely run out of financial resources supporting an expansion of 86 terrabytes / day (assuming 1kb minimum new data per write), far faster than you would ever max out the capabilities of the technology. It's more a finance/business exercise to determine if revenue inflows would actually meet, cover, or exceed revenue outflows on a project like this. You can always make the technology scale and meet the demand.