4 ms·

We chose elixir in 2016 for our product and never regretted it. Apart from application bugs we’ve caused, it just keeps chugging along and working beautifully.

by atonse 5y ago

We chose elixir in 2016 for our product and never regretted it. Apart from application bugs we’ve caused, it just keeps chugging along and working beautifully. It’s never once been the bottleneck in our stack. Finding people has not been tough at our scale, and they’ve been good quality. We are just scratching the surface with OTP. And between that and the data processing tools, we know we can handle pretty much any scale. Deployment is nearly as crude as it was 5 years ago though. We use distillery and edeliver. We are moving to docker containers soon. We use Exq for background jobs and have processed 75 million in a year without a sweat (most of that in the last two months). We handle 55 million requests per month on t3.mediums. We are looking to switch from Exq to Oban but I’m nervous about whether Oban’s Postgres based architecture will handle it. It’ll probably be totally fine.

morbicer 5y ago
55 / 2.62 (millions of seconds in a month) ~= 20reqs / second That's not very impressive, you could achieve that easily on something deemed "slow" like Django or RoR But I guess your load is not constant. How many reqs/sec can it handle in the peak time?
- jcelerier 5y ago
  > 55 / 2.62 (millions of seconds in a month) ~= 20reqs / second I'm pretty sure Java 1.0 running on an Amiga could handle that...
  - airhead969 5y ago
    I bet even 8051 + 10 Mb ISA 10BASE2.
  - deleted 5y ago
    [deleted]
- liveoneggs 5y ago
  to be fair t3.medium is very small for a production deployment ($30/month on-demand) so I read this as having cycles to spare
- atonse 5y ago
  Yes it's not constant. We have a handful of bursts (tens of thousands of SMS conversations happening at 9am every day that complete in about an hour, for each SMS we send out, we get back 3 status callback webhooks from Twilio). And we've had a couple "mass" message sends where it did take a few hours to send 1.6 million messages. But again, that was due to our twilio limits and not elixir. We had to throttle our jobs when Twilio started saying their queue for our number was full. I just checked (because we now process most jobs in the background), we have gone as high as about 500 jobs per second in exq on that same machine (and each job typically runs anywhere from 5 to 15 SQL queries). The bottleneck is actually the size of our RDS instance. Not complaining about postgres, just saying elixir hasn't been the bottleneck for that amount. Every hour we process CSV files with 30,000 records, etc. We only started caching things about a month ago. I have been very lazy about that since it just didn't make any difference until recently. Again, the point I'm trying to make is that using elixir has been positively and utterly boring in the best way possible. No drama whatsoever. We've spent the overwhelming majority of any of the time related to this looking at postgres. Not even complaining about Postgres either, it's more a conversation of what's the right RDS size and IO for our traffic. This stack has caused us very little drama.
pselbert 5y ago
Running 75 million jobs a year would be totally fine. Depending on the configuration and load Oban easily runs thousands of jobs a second, and 75 million breaks down to less than 3 a second. I’d argue it’s worth the switch, but I’m highly biased!
- deleted 5y ago
  [deleted]
- atonse 5y ago
  Thank you! I've been meaning to ask Oban people about this. Sadly, while Exq has been rock solid for us the last few years, Oban seems to have a ton of active development and features we wholeheartedly want to adopt. Just haven't had the time to run tests :) Our workload is highly bursty. We queue about 20k jobs in the morning, which themselves queue another 100k-150k jobs. Then hourly we queue additional ones. I'm concerned about the Postgres overhead of inserting thousands of rows for jobs, each with potentially their own transaction. With Oban, we can actually get rid of Redis in our stack (currently using it for Exq)

by atonse 5y ago

morbicer 5y ago

55 / 2.62 (millions of seconds in a month) ~= 20reqs / second That's not very impressive, you could achieve that easily on something deemed "slow" like Django or RoR But I guess your load is not constant. How many reqs/sec can it handle in the peak time?

jcelerier 5y ago
> 55 / 2.62 (millions of seconds in a month) ~= 20reqs / second I'm pretty sure Java 1.0 running on an Amiga could handle that...
- airhead969 5y ago
  I bet even 8051 + 10 Mb ISA 10BASE2.
- deleted 5y ago
  [deleted]
liveoneggs 5y ago
to be fair t3.medium is very small for a production deployment ($30/month on-demand) so I read this as having cycles to spare
atonse 5y ago
Yes it's not constant. We have a handful of bursts (tens of thousands of SMS conversations happening at 9am every day that complete in about an hour, for each SMS we send out, we get back 3 status callback webhooks from Twilio). And we've had a couple "mass" message sends where it did take a few hours to send 1.6 million messages. But again, that was due to our twilio limits and not elixir. We had to throttle our jobs when Twilio started saying their queue for our number was full. I just checked (because we now process most jobs in the background), we have gone as high as about 500 jobs per second in exq on that same machine (and each job typically runs anywhere from 5 to 15 SQL queries). The bottleneck is actually the size of our RDS instance. Not complaining about postgres, just saying elixir hasn't been the bottleneck for that amount. Every hour we process CSV files with 30,000 records, etc. We only started caching things about a month ago. I have been very lazy about that since it just didn't make any difference until recently. Again, the point I'm trying to make is that using elixir has been positively and utterly boring in the best way possible. No drama whatsoever. We've spent the overwhelming majority of any of the time related to this looking at postgres. Not even complaining about Postgres either, it's more a conversation of what's the right RDS size and IO for our traffic. This stack has caused us very little drama.

pselbert 5y ago

Running 75 million jobs a year would be totally fine. Depending on the configuration and load Oban easily runs thousands of jobs a second, and 75 million breaks down to less than 3 a second. I’d argue it’s worth the switch, but I’m highly biased!

deleted 5y ago
[deleted]
atonse 5y ago
Thank you! I've been meaning to ask Oban people about this. Sadly, while Exq has been rock solid for us the last few years, Oban seems to have a ton of active development and features we wholeheartedly want to adopt. Just haven't had the time to run tests :) Our workload is highly bursty. We queue about 20k jobs in the morning, which themselves queue another 100k-150k jobs. Then hourly we queue additional ones. I'm concerned about the Postgres overhead of inserting thousands of rows for jobs, each with potentially their own transaction. With Oban, we can actually get rid of Redis in our stack (currently using it for Exq)