paulasmuth

16 ms·

91.
by paulasmuth 11y ago
Have a look at Google's MinHash algorithm. While it's a probabilistic solution, You can run it as a mapreduce and will at no point need to have the full data set in memory/on a single machine. So it does scale pretty well.
92.
by paulasmuth 11y ago
How do you determine if I am flooding your service with random data? My understanding is that one of your main ideas is to encrypt the data so I am honestly wondering where you would even start to check if a user was randomly flooding you o
93.
by paulasmuth 11y ago
I am not sure I understand the pricing and/or the service. What are your guarantees about dropping or not dropping keys? Obviously you can't sell unlimited memcache space for 20$/mo and you must protect against customers exha
94.
by paulasmuth 12y ago
Shameless plug: This looks quite similar to FnordMetric, which also supports labels/multi dimensional time series, is StatsD wire compatible and supports SQL as a query language (so you won't have to learn yet another DSL)
95.
by paulasmuth 12y ago
Sounds similar to FnordMetric ( http://fnordmetric.io/chartsql ) which also supports dimensional timeseries data. Major differences between Atlas and FnordMetric on first sight: - SQL based query and charting frontend (ChartS
96.
by paulasmuth 12y ago
You might also be interested FnordMetric ChartSQL ( http://fnordmetric.io ) which is a SQL-based graphite competitor.
97.
by paulasmuth 12y ago
You didn't mistunderstand the purpose (However it's not an explicit goal of FnordMetric to substitute chart.io) Here is an example that connects to a MySQL database on localhost: https://github.com/paulasmuth/
98.
by paulasmuth 12y ago
It's actually fairly straightforward, you can check out the code here: https://github.com/paulasmuth/fnordmetric/tree/master/fnordm...
99.
by paulasmuth 12y ago
> 1. Does IMPORT TABLE mean that it has to copy data from MySQL Database into its own storage or does it merely mean it will connect to the MySQL db and query it directly? It will only connect to the MySQL database and won't copy an
100.
by paulasmuth 12y ago
Yes, you can use FnordMetric to create charts from data in your existing database. Have a look at this documentation page: http://fnordmetric.io/documentation/chartsql/external_data_s... Here is a simple example f
101.
by paulasmuth 12y ago
I think we need to improve the wording in the documentation. The IMPORT statement only creates a "virtual table", it doesn't actually import any data. As much of the query as possible is pushed down into MySQL/the extern
102.
by paulasmuth 12y ago
However that means you have to write a heap of reptitive glue code (or sed incantations if that's your thing) to mangle your SQL Results into the JSON format your charting tool wants. If you run a lot of ad-hoc queries you have to wast
103.
by paulasmuth 12y ago
That should be fixed if you pull the latest master branch from github. We'll provide binary packages for apt-get and homebrew in the next weeks.
104.
by paulasmuth 12y ago
Here is a working example: https://github.com/paulasmuth/fnordmetric/blob/master/fnordm... And a few bits of documentation here http://fnordmetric.io/documentation/chartsql/exte
105.
by paulasmuth 12y ago
Yes, actually: https://www.gnu.org/philosophy/why-not-lgpl.html
106.
by paulasmuth 12y ago
A "deploy to GCE" button is something I am planning to add in the next weeks.
107.
by paulasmuth 12y ago
FnordMetric allows you to query data from a number of sources. CSV Files are one of them, but there are other backends like MySQL and the built-in statsd server. So ideally there is no "middleware step": you generate charts using
108.
ChartSQL: Create Charts and Dashboards from SQL(fnordmetric.io)
271 points by paulasmuth 12y ago|72 comments
109.
by paulasmuth 12y ago
If you set up your tests up correctly they should test the exact binary or shared library that is deployed to production and not some test-specific build.
110.
by paulasmuth 12y ago
Yes, you are right, this is not a binary classification and it really depends on the perspective. What I was trying to say is that - on some levels - the two approaches are almost antithetical.
111.
by paulasmuth 12y ago
> so writing a unikernel like this allows you to select exactly as much as you need for your particular service. I agree this is a big upside of the authors approach. Less dependencies lead to fewer problems caused by external/ups
112.
by paulasmuth 12y ago
No, this is fundamentally different from docker. Docker/cgroups/namespaces allow you to isolate multiple applications running in userspace on the same kernel to a very high degree. This gets rid of isolation and multiprocessing al
113.
by paulasmuth 12y ago
One could argue that threading is not strictly necessary to utilize a multicore machine. A lot of applications (think web apps) can be implemented just fine as a single thread and then scaled up by increasing the number of processes (or the
114.
by paulasmuth 12y ago
This sounds like an awesome and fun project to hack on! However, optimizing around context switches and task preemptions is something you would usually do if your application is actually bound by IO/context switching, is extremely late
115.
by paulasmuth 12y ago
I see your point, however I have to politely disagree: The naive solution to the problem you described would be to simply broadcast every email to every possible receiver. And in fact, I believe this is how bitcoin actually works (but I am
116.
by paulasmuth 12y ago
Can you explain why you think SMTP is not "fully peer-to-peer"?
117.
by paulasmuth 12y ago
Mh, I feel stupid now. In which ways other than relying on DNS and IP is "email" centralized?
118.
by paulasmuth 12y ago
All your points seem to focus on DNS. Nobody would argue that DNS is a decentralized system. And in fact neither is IP. As far as I understand nothing in SMTP makes it inherently more centralized than any piece of software that relies on IP
119.
by paulasmuth 13y ago
We will open source our CF-based recommendation engine in the next days. We use the code in production (on a site with a few hundred million views a month) with up to 80GB datasets for item<->item CF. The engine generates thousands of
120.
by paulasmuth 13y ago
Could you expand on how prediction.io would handle a real world data set containing a few million items/users? How long would it take to generate a single user<->user recommendation at this scale? Does prediction.io require that

More ›