vtuulos

21 ms·

1. 151.▲
   by vtuulos 13y ago

   Btw, if you would be interested in hacking Deliroll and other related things, feel free to contact me - my email is on the slides. We are hiring.
2. 152.▲
   by vtuulos 13y ago

   Matrices are read-only. The set of matrices is updated once a day to include the latest data. The data format is a collection of pre-aggregated row and column vectors, encoded with variable length integers and run-length encoding. I should
3. 153.▲
   by vtuulos 13y ago

   660GB was just a small benchmark. The real thing uses more than a petabyte of raw data. Pandas uses NumPy internally. You could use Deliroll as a replacement for NumPy in Pandas to get a nice interactive environment for amounts of data that
4. 154.▲
   by vtuulos 13y ago

   I started with Scipy Sparse. I ditched it for a custom solution so I could support variable-length integers, run-length encoding etc.
5. 155.▲
   by vtuulos 13y ago

   Reveal.js https://github.com/hakimel/reveal.js/
6. 156.▲
   by vtuulos 13y ago

   Yes, definitely. We might be able to open-source some parts of the system eventually as well.
7. 157.▲
   by vtuulos 13y ago

   Author here: i just checked this morning - we have 100B+ rows in our deliroll matrices, hosted on a single machine. I'm happy to answer any questions.
8. 158.▲
   by vtuulos 13y ago

   one of the biggest benefits of HLL is that if you are only interested in set cardinalities, you don't have to store the original items at all, just a small HLL data structure which can be used to compute set unions, and with some cavea
9. 159.▲
   by vtuulos 13y ago

   Thanks! Our approach supports both discrete and continuous values. It is mainly optimized for the use case where you want to aggregate continuous variables over discrete filters.
10. 160.▲
    by vtuulos 13y ago

    Author here: Fortunately HN works even in the Finnish countryside. I am happy to answer any questions.
11. 161.▲
    by vtuulos 13y ago

    Sorry, not yet. I hope to be able to make more information available soon. The query language is very straightforward. More interestingly, this approach makes it easy to implement various algorithms for machine learning / data mining,
12. 162.▲
    by vtuulos 13y ago

    Numba was just the fastest way to get it working. LLVM(py) is very low-level. C is still an option but Numba made interfacing with a no-brainer.
13. 163.▲
    by vtuulos 13y ago

    Exactly - encoding is the key. I just gave a presentation last week at the SF Python Meetup about how AdRoll uses a single server to query terabytes of compressed data with sub-minute latencies, using a system that is implemented in Pytho
14. 164.▲
    Customizable analytics for your GitHub repositories(blog.bitdeli.com)

    2 points by vtuulos 14y ago|0 comments
15. 165.▲
    by vtuulos 14y ago

    Unix command-line tools are seriously undervalued for processing data, even big data. Need parallelization? Try xargs -P on a multi-core machine.
16. 166.▲
    Build Data-Driven Email Campaigns in 5 Easy Steps(blog.bitdeli.com)

    3 points by vtuulos 14y ago|0 comments
17. 167.▲
    by vtuulos 14y ago

    Great library! I know that a number of companies have created a wrapper of their own exactly for this purpose. We could contribute integration to https://bitdeli.com for those who want to build their own analytics. Is it enough to add a n
18. 168.▲
    by vtuulos 14y ago

    Shameless plug: https://bitdeli.com lets you put theory into practice - you can build your own web analytics in plain and simple Python. Feel free to ping me if you need help getting started (or a longer trial :)
19. 169.▲
    by vtuulos 14y ago

    Feeding data from SnowPlow to Bitdeli sounds like a great idea. 1. It does, although it is an undocumented feature. You can send a list of events instead of a single JSON object to the Events API. 2. Bitdeli doesn't need an event type but i
20. 170.▲
    by vtuulos 14y ago

    The actual analytics work is done by the Python scripts which are open-source. See https://github.com/bitdeli or sign up to Bitdeli where you can see them in action. The backend takes care of the plumbing: receiving and persisting data, s
21. 171.▲
    by vtuulos 14y ago

    Btw, for more details about what powers Bitdeli, see http://weusethat.com/bitdeli/ The backend is mostly in Erlang. The frontend is made of Backbone, D3, Bootstrap.
22. 172.▲
    by vtuulos 14y ago

    With Mixpanel we grab everything back to the beginning of time. How much data is made available for a dashboard (card) depends on your plan. See details here: https://bitdeli.com/docs/datasources.html
23. 173.▲
    by vtuulos 14y ago

    Thanks for the feedback! + Technically a live demo with a static dataset would be doable. [adds to Trello] + We think that $49 is reasonable given how powerful and flexible the service is. We can extend your trial if you are not convinced :
24. 174.▲
    by vtuulos 14y ago

    Yes, definitely. The biggest issue with importing data from Google Analytics is that they have a pretty low quota for data export, so it is hard/slow to get all historical data out from GA.
25. 175.▲
    by vtuulos 14y ago

    Hi! We have a 14 day free trial - no obligation. Additional information can be best found at our documentation: https://bitdeli.com/docs/
26. 176.▲
    by vtuulos 14y ago

    Hi HN! We just (re)launched today. All feedback would be greatly appreciated. I'm happy to answer any questions here.
27. 177.▲
    by vtuulos 14y ago

    I feel your pain. I started http://discoproject.org back in 2007, trying to make MapReduce more easily approachable. If vanilla MapReduce is too low-level for you, you might want to take a look at http://bitdeli.com which provides a clo
28. 178.▲
    by vtuulos 15y ago

    Thanks - we will be adding more examples in the near future. Creating a visualization with Bitdeli is not hard at all if you know JavaScript and little Python - see a tutorial here: http://blog.bitdeli.com/2012/03/14/how-to-visualize-geogr
29. 179.▲
    by vtuulos 15y ago

    Btw, remember to check out our in-browser editor. It lets you run arbitrary Python code with live data on the fly. Click "fork script" on any script page, e.g. here https://bitdeli.com/scripts/94/convert-bart-xml-to-json I (Ville, one of
30. 180.▲
    by vtuulos 15y ago

    Actually the largest MapReduce cluster at Nokia runs Disco. Feel free to contact me at ville.h.tuulos at nokia dot com if you want to give it a try.