5 ms·

Congrats on the launch! I've actually found myself using Convore more often than I thought I would. Particularly because I'm often in places that block irc. Ha

Congrats on the launch! I've actually found myself using Convore more often than I thought I would. Particularly because I'm often in places that block irc. Have you looked at distributed counting in Cassandra for your counting needs yet[0][1]? Great info on its development and use at Twitter. It seems like you too have lots of interesting things to count. Your initial choice of Postgres for everything was a bit interesting. Your problem seems like a perfect fit for a hybrid solution (which you are already implementing by way of redis) that I think more and more companies will come to embrace by hook or by crook. Continued success! [0]General nosql at Twitter and a fair bit on Cassandra: http://www.infoq.com/presentations/NoSQL-at-Twitter-by-Ryan-King http://www.infoq.com/presentations/NoSQL-at-Twitter-by-Ryan-... [1]Specifics on Rainbird, the counting system at twitter built on Cassandra: http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011 http://www.slideshare.net/kevinweil/rainbird-realtime-analyt... [cross post from the blog]

ericflo 16y ago
Yeah, I've been following that work for quite a while, it's really impressive! I'm a big fan of Cassandra in general (in fact, I wrote an example application to help teach beginners about it: https://github.com/ericflo/twissandra https://github.com/ericflo/twissandra). The decision to go with PostgreSQL over Cassandra (or another distributed system like Riak or HBase) was simply because it gave us the most flexibility to change our product quickly while we operate at low scale. And if I'm honest with myself, we're operating at very low scale right now. In the future as we scale up, Cassandra's distributed counters will be one of the first places we look.
- saurik 16y ago
  Are you willing to say how many simultaneous users you are handling at this point? (I obviously totally understand if you aren't.)
  - ericflo 16y ago
    Unfortunately I can't give out any exact numbers :(
    - saurik 16y ago
      I realize you may also not want to answer this, and that's also alright, but I figure I may as well ask, as I really am curious (and maybe someone else will answer from the perspective of their company): why not? Whenever anyone has asked me for a stat on Cydia, whether it be active devices over some time range, daily revenue, costs related to <insert-subproject-here>, or what have you, I tend to break out a quick SQL query and provide an exact answer (assuming the question is of something I an measure: not all are); is this stupid of me? I've noticed a ton of companies refuse to disclose numbers, and I've always assumed that the result will be that anyone listening will just assume "ok, so almost none then" unless you give them a good answer and back up how you calculated it, but does this actually "hurt the cause"? (I do not have the benefits of the years if startup experience that you have access to by being a part of Y Combinator, so I try to get advice whenever I can ;P.)
      - axod 16y ago
        The top 20 public groups topped out at about 250 users last night, add on a bit for the other public groups and a little for private groups. Maybe 400 concurrent users max so far? Just my guess, but I'd be surprised if it's orders of magnitude different.
- foobarbazetc 16y ago
  Just FYI: we run a very large web service with purely PostgreSQL, and have no need for sharding or anything of the sort. Scale up first. Use redis or memcached to store 'overview' stats. If you lose those stats, recalc from Pg.
  - ericflo 16y ago
    That's exactly the approach we're taking. We're not going to spend any time prematurely worrying about various scaling things until we can forecast that it's legitimately going to be a problem.
    - foobarbazetc 16y ago
      Great to hear. :) While we're here, here are some PG tips: 1. Make sure your PG install is fully optimized. I've seen countless postgresql.conf's at major companies with no opts what so ever. Take a look at this book: http://www.2ndquadrant.com/books/postgresql-9-0-high-performance/ http://www.2ndquadrant.com/books/postgresql-9-0-high-perform... . 2. Make sure you're using PG 9+. 3. Use pgbouncer for connection pooling and try to write your app so you can use 'pool_mode = transaction'. 4. Please don't run PG virtualized.
      - ericflo 16y ago
        We're good on all fronts except 4, and the transaction pool mode :)
saurik 16y ago
Give that you know of IRC (and were inconvenienced that it was blocked), have you tried irccloud.com? The people who did that are also on HN and it has worked out great for the friends of mine I've gotten to use it so far (I have a lot of friends with inconsistent network connections or who are not very technical that I wanted to be able to access my IRC server).
- siculars 16y ago
  I haven't. Just signed up for a beta account. Looks very cool though.

ericflo 16y ago

Yeah, I've been following that work for quite a while, it's really impressive! I'm a big fan of Cassandra in general (in fact, I wrote an example application to help teach beginners about it: https://github.com/ericflo/twissandra https://github.com/ericflo/twissandra). The decision to go with PostgreSQL over Cassandra (or another distributed system like Riak or HBase) was simply because it gave us the most flexibility to change our product quickly while we operate at low scale. And if I'm honest with myself, we're operating at very low scale right now. In the future as we scale up, Cassandra's distributed counters will be one of the first places we look.

saurik 16y ago
Are you willing to say how many simultaneous users you are handling at this point? (I obviously totally understand if you aren't.)
- ericflo 16y ago
  Unfortunately I can't give out any exact numbers :(
  - saurik 16y ago
    I realize you may also not want to answer this, and that's also alright, but I figure I may as well ask, as I really am curious (and maybe someone else will answer from the perspective of their company): why not? Whenever anyone has asked me for a stat on Cydia, whether it be active devices over some time range, daily revenue, costs related to <insert-subproject-here>, or what have you, I tend to break out a quick SQL query and provide an exact answer (assuming the question is of something I an measure: not all are); is this stupid of me? I've noticed a ton of companies refuse to disclose numbers, and I've always assumed that the result will be that anyone listening will just assume "ok, so almost none then" unless you give them a good answer and back up how you calculated it, but does this actually "hurt the cause"? (I do not have the benefits of the years if startup experience that you have access to by being a part of Y Combinator, so I try to get advice whenever I can ;P.)
    - axod 16y ago
      The top 20 public groups topped out at about 250 users last night, add on a bit for the other public groups and a little for private groups. Maybe 400 concurrent users max so far? Just my guess, but I'd be surprised if it's orders of magnitude different.
foobarbazetc 16y ago
Just FYI: we run a very large web service with purely PostgreSQL, and have no need for sharding or anything of the sort. Scale up first. Use redis or memcached to store 'overview' stats. If you lose those stats, recalc from Pg.
- ericflo 16y ago
  That's exactly the approach we're taking. We're not going to spend any time prematurely worrying about various scaling things until we can forecast that it's legitimately going to be a problem.
  - foobarbazetc 16y ago
    Great to hear. :) While we're here, here are some PG tips: 1. Make sure your PG install is fully optimized. I've seen countless postgresql.conf's at major companies with no opts what so ever. Take a look at this book: http://www.2ndquadrant.com/books/postgresql-9-0-high-performance/ http://www.2ndquadrant.com/books/postgresql-9-0-high-perform... . 2. Make sure you're using PG 9+. 3. Use pgbouncer for connection pooling and try to write your app so you can use 'pool_mode = transaction'. 4. Please don't run PG virtualized.
    - ericflo 16y ago
      We're good on all fronts except 4, and the transaction pool mode :)

saurik 16y ago

Give that you know of IRC (and were inconvenienced that it was blocked), have you tried irccloud.com? The people who did that are also on HN and it has worked out great for the friends of mine I've gotten to use it so far (I have a lot of friends with inconsistent network connections or who are not very technical that I wanted to be able to access my IRC server).

siculars 16y ago
I haven't. Just signed up for a beta account. Looks very cool though.