13 ms·

Elixir at Ramp

175 points by judicious 3y ago

sb8244 3y ago
This is from 2021. I'd be curious if Elixir is still in use, or has expanded at all. Side note: Pablo's blog posts are always quite good.
- thibaut_barrere 3y ago
  I also wondered about this but could not find anything definitive on that. They seem to be a bit "stealth" about the technologies in use at least based on reading their job offers: https://jobs.ashbyhq.com/ramp/6a5108f4-8605-4a67-9446-ea05f197694c https://jobs.ashbyhq.com/ramp/6a5108f4-8605-4a67-9446-ea05f1... > Proficiency in Python, Go, or Java
  - paulbgd 3y ago
    I’m not sure why, but the internship role seems to be the only one listed that has an accurate list of our tech: https://jobs.ashbyhq.com/ramp/29663a4b-c457-4a38-bbdf-069f189b20eb?departmentId=e9877d64-61b1-4b37-8518-65af0431cd09 https://jobs.ashbyhq.com/ramp/29663a4b-c457-4a38-bbdf-069f18... The article goes into a little more detail too, we haven’t significantly changed anything in the last 2 years.
- judicious 3y ago
  Yeah, I’m definitely curious too. From what I know Ramp is primarily a Python shop on the backend. I just love to see functional programming being used in industry and seeing the thought process behind it. Also, I agree. I’m a big fan of Pablo’s posts.
- paulbgd 3y ago
  Yep it’s still being used as our card authorizer service! The features have expanded, although the primary backend monolith is still in python.
  - judicious 3y ago
    Do you foresee a larger scale migration to Elixir in the future? Or would you say that Elixir lends itself to some services and Python others?
    - jstummbillig 3y ago
      And if so: Why is that? Sometimes I read a glowing review and wonder "Why not Elixir everything?" What's the catch?
      - oldpersonintx 3y ago
        [dead]
      - srpablo 3y ago
        Author here; I have a lot of other posts in my personal blog about this, but: the current trends in VC-backed tech companies are about minimizing risk and following fashion, rather than any technical merit or experimentation. Said another way: if an Elixir company dies, it's "damn, shouldn't have picked Elixir!" If a Python company dies, it's "startups are hard," with no investigation behind what Python cost you. I go into it a bit here https://morepablo.com/2023/05/where-have-all-the-hackers-gone.html https://morepablo.com/2023/05/where-have-all-the-hackers-gon... and here https://morepablo.com/2023/06/creatives-industries.html https://morepablo.com/2023/06/creatives-industries.html Elixir has real technical downsides too, but honestly they never come up when someone advocates against it. And this is fine, building companies and engineering culture is a social game at the end of the day.
        thibaut_barrere 3y ago
        Could you maybe share your perception of the technical downsides of Elixir?
        ironchef 3y ago
        The libraries out there lack the breadth and maturity of some of the other ecosystems (as a simple example). Or at least they did for some of my corners of the world.
- srpablo 3y ago
  Thanks for the kind words! Another comment mentions it, but last I checked Elixir is still powering the Authorization service. You might be interested in a "making of" post I wrote on my personal blog, going into the intentions and crafting of this https://morepablo.com/2023/03/technical-storytelling-making-of-elixir-at-ramp.html https://morepablo.com/2023/03/technical-storytelling-making-...
  - judicious 3y ago
    I loved reading the “making of” post. I’m very interested in getting into technical writing and your post, really helped me understand your thought process behind “Elixir at Ramp”.
- innocentoldguy 3y ago
  We switched from Java to Elixir and the change has doubled our productivity. Elixir is the most pleasant language I've every worked with (30+ years in software engineering), so I recommend giving it a try.
colesantiago 3y ago
(2021) Unfortunately Brex doesn't use Elixir anymore https://medium.com/brexeng/building-backend-services-with-kotlin-7c8410795e4b https://medium.com/brexeng/building-backend-services-with-ko...
- judicious 3y ago
  Kotlin, definitely an interesting choice for backend development. But I love to see idiosyncratic language choices being made.
  - vvpan 3y ago
    Doordash uses Kotlin for backend as well. I suppose they want FE/BE to be in the same language. And when your main product is a native app Kotlin starts to make sense.
    - ceras 3y ago
      Neither the iOS app nor the web app use Kotlin at DoorDash. Kotlin was picked for other reasons (consistency with the Android app wasn't mentioned): https://doordash.engineering/2021/05/04/migrating-from-python-to-kotlin-for-our-backend-services/ https://doordash.engineering/2021/05/04/migrating-from-pytho...
  - brink 3y ago
    Having spent considerable time in both Java and Elixir, I would also choose Kotlin (or Java) over Elixir as a backend language. I worked in Elixir for over a year, and frankly was quite disappointed with it. Elixir has a lot of good ideas, but falls short in some crucial points in execution. Relying on shapes instead of types to identify results / objects, weird syntax choices (You use both <%= %> and {} to interpolate in phoenix templates, depending on where you are), no ability to write custom guards, a lot of language concept overlap - for example behaviors / protocols should be one implementation instead of two separate ideas.. Elixir is an okay language, but I think it's just a fad, not good enough to have staying power. I think a better language written on the BEAM will come along and be the language that Elixir should have been. Just my personal opinion.
Dowwie 3y ago
Ramp and Divvy (Bill), both in the expense management space, chose Elixir for a portion of their payment systems.
sesm 3y ago
When I worked with Erlang outside of Ericsson, I’ve heard the following from our tech lead: OTP built-in distributed process and failover mechanisms assume much lower latencies than usually exist between datacenter VMs and the underlying protocols consume a lot of bandwidth. I didn’t dig deep enough to verify this, but I can confirm that typical Ericsson hardware is several boards connected with high bandwidth ‘backplane’ and OTP failover was designed for this environment. VMs communicating through LAN is a different environment, and VMs living in different data centers are very far from this. I wonder if anyone else tried using OTP outside of Ericsson and what is their experience.
- throwawaymaths 3y ago
  You should probably reach for something like ra for anything but trivial stuff. If latency is not a concern, global module is probably fine but double check the quorum membership.
  - rhodin 3y ago
    ra for those interested: https://github.com/rabbitmq/ra https://github.com/rabbitmq/ra ("A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.") Used by RabbitMQ for quorum queues + Khepri (the new metadata store)
- toast0 3y ago
  Welp, my browser ate my longer response. But in short, when I was at WhatsApp, dist worked fine over WANs, I think the WA contributed pg addresses the issues we saw with pg2. It was obviously not ideal, or normal, but sometimes our WAN connections would get bandwidth limited due to congestion or other issues, and things would generally keep going without unexpected problems: replication got behind when the change stream was more than available bandwidth and there's no separation of dist traffic, so if replication has minutes or hours of delay, so does any other traffic in that direction, so interactive traffic will timeout, net_adm ping will take minutes or hours, etc. I guess technically the connection ticks must not quite suffer the same queuing, because the connections would not be severed by tick timeouts. The other issue related to WAN that we'd see is when you start a mnesia node, it loads all of the tables from peers if peers have copies of the tables, but out of the box it doesn't have a concept of LAN vs WAN peers, so it may load your half a terrabyte or whatever worth of data from WAN instead of LAN. Loading all that data from LAN might be undesirable too, but that's another story; you could probably maybe come up with a way to store and later replicate changelogs for offline mnesia peers, but that doesn't come out of the box, and WA didn't build it either. The community at large often reported maximum dist cluster sizes that were way less than what we ran in production, and comments about WAN distribution that didn't match our experience either. shrug It does help to have a very solid private network between nodes though. Dist in general, but mnesia specifically, don't like for nodes to disappear. pg2 has some quirks too, but pg resolves them, afaik, as I mentioned earlier. Some of pg2's issues were rooted in global locks, you've got to be careful about those if node lists are rapidly changing and there's contention on the locks.
freedomben 3y ago
> My hot take about most dynamic languages is that they are a poor fit for startups who have intentions of being long-term businesses: you've created an environment that's optimized for your founding engineers to build something quickly in the first 7 months, but the price is a set of recurring obstacles which your engineers will pay down over the next 7 years. I've been saying this for years, so it feels really good to the confirmation bias to see it said by someone else! Though, the language and frameworks definitely matter, but you can easily write short-term-over-long-term code in any language/framework. Founding engineers tend to be the worst as well, because they love and sometimes only ever experienced greenfield development, and they are trying to move fast. That's not a bad thing as without that the startup may die on the vine, but it is something that I wish more founding engineers would be cognisant of, because little things (json schemas, and unit/integration tests for example!) can make a big difference. Elixir (and Phoenix) really do get the best of both worlds. It's highly productive like Rails, but the built-in way of doing things will give you validations on your data in and there are plenty of tools like Ecto schemas that can be used anywhere and everywhere the data matters. Tests are also first-class citizens and have to be actively ignored to avoid them. The one area where IMHO the jury is still out is LiveView. It takes a lot of battle-hardened approaches of React, and my initial feelings on it are very good. However I've not yet had a natural organization structure emerge with it, so I'm not sure how maintainable it will be in the future for people that didn't write it. Anybody have experience with that?
- imachine1980_ 3y ago
  does elixir have type checking like typescript does ?( you can add types ?)
  - hmmokidk 3y ago
    The hottest news in the Elixir community is that it is currently experimenting with set theoretic type / gradual typing. Some french PHD is working on it. It is past the RND phase and in development at the moment. It is highly experimental and not guaranteed to ship with the next version of Elixir. Right now the idea is to see if it offers any value in the first place. There are a lot of statements about what types offer. What the Elixir team wants to do is actually experiment to see if there's a benefit. Which is a really neat thing for a lang to do...to actually experiment like that is incredible. As for right now, in a sense yes. It has this thing called Dialyzer - https://www.erlang.org/doc/apps/dialyzer/dialyzer_chapter.html https://www.erlang.org/doc/apps/dialyzer/dialyzer_chapter.ht... which is a static analysis tool. This offers some non runtime type guarantees. But it is still a dynamically typed language. And the community is split on it IME. The errors are cryptic and there are some other issues. There's also Norm. There are some things you can do to guarantee types during runtime, like match a function to a type. So when called with a string the implementation is different than with a boolean. So there are some run time type guarantees. example of matching (return a diff string if bool or if nil): def do_thing(n) when is_bool(n): "some boolean" def do_thing(n) when is_nil(n): "nil" There's also the concept of a struct which is like a typed object / map / key value pair thing.
hellozomo 3y ago
As a user, Ramp's web interface is absolute trash. I've submitted reimbursement requests only to get an error message and then have them go through later, creating duplicates. CC transactions queue up, requiring a response and there's no way to go through them efficiently. The UI pops up a weird modal and is horrible to use. It feels like a legacy enterprise application and it's basically a new product. I wish my company didn't use it.

175 points by judicious 3y ago

This is from 2021. I'd be curious if Elixir is still in use, or has expanded at all. Side note: Pablo's blog posts are always quite good.

thibaut_barrere 3y ago
I also wondered about this but could not find anything definitive on that. They seem to be a bit "stealth" about the technologies in use at least based on reading their job offers: https://jobs.ashbyhq.com/ramp/6a5108f4-8605-4a67-9446-ea05f197694c https://jobs.ashbyhq.com/ramp/6a5108f4-8605-4a67-9446-ea05f1... > Proficiency in Python, Go, or Java
- paulbgd 3y ago
  I’m not sure why, but the internship role seems to be the only one listed that has an accurate list of our tech: https://jobs.ashbyhq.com/ramp/29663a4b-c457-4a38-bbdf-069f189b20eb?departmentId=e9877d64-61b1-4b37-8518-65af0431cd09 https://jobs.ashbyhq.com/ramp/29663a4b-c457-4a38-bbdf-069f18... The article goes into a little more detail too, we haven’t significantly changed anything in the last 2 years.
judicious 3y ago
Yeah, I’m definitely curious too. From what I know Ramp is primarily a Python shop on the backend. I just love to see functional programming being used in industry and seeing the thought process behind it. Also, I agree. I’m a big fan of Pablo’s posts.
paulbgd 3y ago
Yep it’s still being used as our card authorizer service! The features have expanded, although the primary backend monolith is still in python.
- judicious 3y ago
  Do you foresee a larger scale migration to Elixir in the future? Or would you say that Elixir lends itself to some services and Python others?
  - jstummbillig 3y ago
    And if so: Why is that? Sometimes I read a glowing review and wonder "Why not Elixir everything?" What's the catch?
    - oldpersonintx 3y ago
      [dead]
    - srpablo 3y ago
      Author here; I have a lot of other posts in my personal blog about this, but: the current trends in VC-backed tech companies are about minimizing risk and following fashion, rather than any technical merit or experimentation. Said another way: if an Elixir company dies, it's "damn, shouldn't have picked Elixir!" If a Python company dies, it's "startups are hard," with no investigation behind what Python cost you. I go into it a bit here https://morepablo.com/2023/05/where-have-all-the-hackers-gone.html https://morepablo.com/2023/05/where-have-all-the-hackers-gon... and here https://morepablo.com/2023/06/creatives-industries.html https://morepablo.com/2023/06/creatives-industries.html Elixir has real technical downsides too, but honestly they never come up when someone advocates against it. And this is fine, building companies and engineering culture is a social game at the end of the day.
      - thibaut_barrere 3y ago
        Could you maybe share your perception of the technical downsides of Elixir?
        ironchef 3y ago
        The libraries out there lack the breadth and maturity of some of the other ecosystems (as a simple example). Or at least they did for some of my corners of the world.
srpablo 3y ago
Thanks for the kind words! Another comment mentions it, but last I checked Elixir is still powering the Authorization service. You might be interested in a "making of" post I wrote on my personal blog, going into the intentions and crafting of this https://morepablo.com/2023/03/technical-storytelling-making-of-elixir-at-ramp.html https://morepablo.com/2023/03/technical-storytelling-making-...
- judicious 3y ago
  I loved reading the “making of” post. I’m very interested in getting into technical writing and your post, really helped me understand your thought process behind “Elixir at Ramp”.
innocentoldguy 3y ago
We switched from Java to Elixir and the change has doubled our productivity. Elixir is the most pleasant language I've every worked with (30+ years in software engineering), so I recommend giving it a try.

colesantiago 3y ago

(2021) Unfortunately Brex doesn't use Elixir anymore https://medium.com/brexeng/building-backend-services-with-kotlin-7c8410795e4b https://medium.com/brexeng/building-backend-services-with-ko...

judicious 3y ago
Kotlin, definitely an interesting choice for backend development. But I love to see idiosyncratic language choices being made.
- vvpan 3y ago
  Doordash uses Kotlin for backend as well. I suppose they want FE/BE to be in the same language. And when your main product is a native app Kotlin starts to make sense.
  - ceras 3y ago
    Neither the iOS app nor the web app use Kotlin at DoorDash. Kotlin was picked for other reasons (consistency with the Android app wasn't mentioned): https://doordash.engineering/2021/05/04/migrating-from-python-to-kotlin-for-our-backend-services/ https://doordash.engineering/2021/05/04/migrating-from-pytho...
- brink 3y ago
  Having spent considerable time in both Java and Elixir, I would also choose Kotlin (or Java) over Elixir as a backend language. I worked in Elixir for over a year, and frankly was quite disappointed with it. Elixir has a lot of good ideas, but falls short in some crucial points in execution. Relying on shapes instead of types to identify results / objects, weird syntax choices (You use both <%= %> and {} to interpolate in phoenix templates, depending on where you are), no ability to write custom guards, a lot of language concept overlap - for example behaviors / protocols should be one implementation instead of two separate ideas.. Elixir is an okay language, but I think it's just a fad, not good enough to have staying power. I think a better language written on the BEAM will come along and be the language that Elixir should have been. Just my personal opinion.

Dowwie 3y ago

Ramp and Divvy (Bill), both in the expense management space, chose Elixir for a portion of their payment systems.

sesm 3y ago

When I worked with Erlang outside of Ericsson, I’ve heard the following from our tech lead: OTP built-in distributed process and failover mechanisms assume much lower latencies than usually exist between datacenter VMs and the underlying protocols consume a lot of bandwidth. I didn’t dig deep enough to verify this, but I can confirm that typical Ericsson hardware is several boards connected with high bandwidth ‘backplane’ and OTP failover was designed for this environment. VMs communicating through LAN is a different environment, and VMs living in different data centers are very far from this. I wonder if anyone else tried using OTP outside of Ericsson and what is their experience.

throwawaymaths 3y ago
You should probably reach for something like ra for anything but trivial stuff. If latency is not a concern, global module is probably fine but double check the quorum membership.
- rhodin 3y ago
  ra for those interested: https://github.com/rabbitmq/ra https://github.com/rabbitmq/ra ("A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.") Used by RabbitMQ for quorum queues + Khepri (the new metadata store)
toast0 3y ago
Welp, my browser ate my longer response. But in short, when I was at WhatsApp, dist worked fine over WANs, I think the WA contributed pg addresses the issues we saw with pg2. It was obviously not ideal, or normal, but sometimes our WAN connections would get bandwidth limited due to congestion or other issues, and things would generally keep going without unexpected problems: replication got behind when the change stream was more than available bandwidth and there's no separation of dist traffic, so if replication has minutes or hours of delay, so does any other traffic in that direction, so interactive traffic will timeout, net_adm ping will take minutes or hours, etc. I guess technically the connection ticks must not quite suffer the same queuing, because the connections would not be severed by tick timeouts. The other issue related to WAN that we'd see is when you start a mnesia node, it loads all of the tables from peers if peers have copies of the tables, but out of the box it doesn't have a concept of LAN vs WAN peers, so it may load your half a terrabyte or whatever worth of data from WAN instead of LAN. Loading all that data from LAN might be undesirable too, but that's another story; you could probably maybe come up with a way to store and later replicate changelogs for offline mnesia peers, but that doesn't come out of the box, and WA didn't build it either. The community at large often reported maximum dist cluster sizes that were way less than what we ran in production, and comments about WAN distribution that didn't match our experience either. shrug It does help to have a very solid private network between nodes though. Dist in general, but mnesia specifically, don't like for nodes to disappear. pg2 has some quirks too, but pg resolves them, afaik, as I mentioned earlier. Some of pg2's issues were rooted in global locks, you've got to be careful about those if node lists are rapidly changing and there's contention on the locks.

freedomben 3y ago

> My hot take about most dynamic languages is that they are a poor fit for startups who have intentions of being long-term businesses: you've created an environment that's optimized for your founding engineers to build something quickly in the first 7 months, but the price is a set of recurring obstacles which your engineers will pay down over the next 7 years. I've been saying this for years, so it feels really good to the confirmation bias to see it said by someone else! Though, the language and frameworks definitely matter, but you can easily write short-term-over-long-term code in any language/framework. Founding engineers tend to be the worst as well, because they love and sometimes only ever experienced greenfield development, and they are trying to move fast. That's not a bad thing as without that the startup may die on the vine, but it is something that I wish more founding engineers would be cognisant of, because little things (json schemas, and unit/integration tests for example!) can make a big difference. Elixir (and Phoenix) really do get the best of both worlds. It's highly productive like Rails, but the built-in way of doing things will give you validations on your data in and there are plenty of tools like Ecto schemas that can be used anywhere and everywhere the data matters. Tests are also first-class citizens and have to be actively ignored to avoid them. The one area where IMHO the jury is still out is LiveView. It takes a lot of battle-hardened approaches of React, and my initial feelings on it are very good. However I've not yet had a natural organization structure emerge with it, so I'm not sure how maintainable it will be in the future for people that didn't write it. Anybody have experience with that?

imachine1980_ 3y ago
does elixir have type checking like typescript does ?( you can add types ?)
- hmmokidk 3y ago
  The hottest news in the Elixir community is that it is currently experimenting with set theoretic type / gradual typing. Some french PHD is working on it. It is past the RND phase and in development at the moment. It is highly experimental and not guaranteed to ship with the next version of Elixir. Right now the idea is to see if it offers any value in the first place. There are a lot of statements about what types offer. What the Elixir team wants to do is actually experiment to see if there's a benefit. Which is a really neat thing for a lang to do...to actually experiment like that is incredible. As for right now, in a sense yes. It has this thing called Dialyzer - https://www.erlang.org/doc/apps/dialyzer/dialyzer_chapter.html https://www.erlang.org/doc/apps/dialyzer/dialyzer_chapter.ht... which is a static analysis tool. This offers some non runtime type guarantees. But it is still a dynamically typed language. And the community is split on it IME. The errors are cryptic and there are some other issues. There's also Norm. There are some things you can do to guarantee types during runtime, like match a function to a type. So when called with a string the implementation is different than with a boolean. So there are some run time type guarantees. example of matching (return a diff string if bool or if nil): def do_thing(n) when is_bool(n): "some boolean" def do_thing(n) when is_nil(n): "nil" There's also the concept of a struct which is like a typed object / map / key value pair thing.

hellozomo 3y ago

As a user, Ramp's web interface is absolute trash. I've submitted reimbursement requests only to get an error message and then have them go through later, creating duplicates. CC transactions queue up, requiring a response and there's no way to go through them efficiently. The UI pops up a weird modal and is horrible to use. It feels like a legacy enterprise application and it's basically a new product. I wish my company didn't use it.