4 ms·

Putting a full power search engine in Ecto

144 points by philippemnoel 2y ago

FrancoisBosun 2y ago
This looks very, very interesting! Good work. My only nitpick is the ligatures. I believe pipelining in Elixir uses the |> operator, but the blog post uses a kind of triangle pointing to the right. Due to my previous exposure to Elixir, I guessed that it must have been |>, but if I hadn’t know, then I would be really confused when I tried to write that in my editor to replicate the code.
- lawn 2y ago
  I personally like some types of ligatures, but I think it's good to not use them when others should read the code.
- brightball 2y ago
  There is some editor plugin that converts it visually to a triangle. I have seen other people use it.
  - arrowsmith 2y ago
    It’s a font, not an editor plugin. Not sure which font specifically is used in the article but an example of a monospace font with ligatures is Fira: https://github.com/tonsky/FiraCode https://github.com/tonsky/FiraCode
  - lvass 2y ago
    prettify-symbols-mode in Emacs.
- sbuttgereit 2y ago
  I agree. I use ligatures in my own coding and so to my eyes, the presentation was very natural... but for someone that doesn't/hasn't I think your point is completely correct. It's better to not use ligatures for publication, such as in this scenario. (Now that I've said that, I better go check and see if I've made this mistake due to just not thinking about it.... hmm.....)
- carrja99 2y ago
  A lot of folks configure their editor to render |> as a rotated triangle.
  - Kamq 2y ago
    Sure, but putting it in a code sample is similar to putting opening/closing quotes in a code sample instead of " It makes it harder for people to copy and paste and play with.
    - dpatterbee 2y ago
      If you copy and paste that triangle you will get "|>".
- mise_en_place 2y ago
  100% agree, it's just jarring to anyone who's developed in Elixir before. It's just like dquote characters on MacOS ("smart quotes")
- LorenzoGood 2y ago
  I, as a new elixir user, was personally confused by this exact thing as well.
conradfr 2y ago
Quite interesting, except the fork aspect.
- Moosieus 2y ago
  Yeah, it’s a non-starter in a lot of ways. I’m working on an alternate implementation that uses fragments, enabled by ongoing updates to ParadeDB’s syntax. Gonna explore all options and see where the cards land.
latch 2y ago
I don't understand why people use Ecto (or ActiveRecord, or...) Back in the day, I'm pretty sure we were using Hibernate and friend because our software was shipped and we wanted it to work with whatever database the client was using. But for a hosted software, what's the point? Not having to know SQL or details about PostgreSQL / the underlying DB ? Apps should be using SQL directly, and for cases where you need dynamic SQL (like, you're where clause is different based on some query string parameters), you can have a low-level query builder (1) (1) I'm not affiliated with it and have never used it, but a good search came up with https://github.com/robconery/moebius https://github.com/robconery/moebius which, at least from the readme, is roughly what I'm talking about.
- isodev 2y ago
  > we wanted it to work with whatever database While this is an option, making an app work with different database backends is not usually why one picks a library like Ecto or ActiveRecord. You see, these frameworks bring a tone of facilities for working with data, adding common solutions so we don't have re-invent the wheel every time. Everything from sanitation of user inputs and query building to creating update statements, validating input data and generating validation messages. Ecto is smart enough to even offer facilities for creating and handling forms based on the provided schema of the data. It also works with virtual data (not backed by a database) and all that can happen in Elixir, the same language as the rest of the application. For me, this has the advantage that I can focus on building what my app is supposed to do. Of course, if you really want or need to, there is always an option to submit your "direct SQL" query, and you will be free to write all the boilerplate needed to handle that.
- prophesi 2y ago
  Ecto isn't an ORM like ActiveRecord or Hibernate. It's a DSL for building queries and you can always drop down into raw SQL if you'd like. Moebius looks like Ecto except now you no longer have schema validations and the like.
  - sph 2y ago
    Yes. It's more of a data mapper and composable DSL over SQL than an "object-relational mapper", since indeed there are no stateful objects in Elixir.
dugmartin 2y ago
If you are looking for a great blog post describing how to build a native full text search engine in Elixir that you can drop into your app: https://culttt.com/2023/03/22/building-a-full-text-search-engine-in-elixir https://culttt.com/2023/03/22/building-a-full-text-search-en... (I’ve no affiliation with the author)
skybrian 2y ago
Context: Ecto [1] seems to be a database layer for Elixir and Elixir is a programming language for Erlang’s virtual machine. [1] https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto

144 points by philippemnoel 2y ago

This looks very, very interesting! Good work. My only nitpick is the ligatures. I believe pipelining in Elixir uses the |> operator, but the blog post uses a kind of triangle pointing to the right. Due to my previous exposure to Elixir, I guessed that it must have been |>, but if I hadn’t know, then I would be really confused when I tried to write that in my editor to replicate the code.

lawn 2y ago
I personally like some types of ligatures, but I think it's good to not use them when others should read the code.
brightball 2y ago
There is some editor plugin that converts it visually to a triangle. I have seen other people use it.
- arrowsmith 2y ago
  It’s a font, not an editor plugin. Not sure which font specifically is used in the article but an example of a monospace font with ligatures is Fira: https://github.com/tonsky/FiraCode https://github.com/tonsky/FiraCode
- lvass 2y ago
  prettify-symbols-mode in Emacs.
sbuttgereit 2y ago
I agree. I use ligatures in my own coding and so to my eyes, the presentation was very natural... but for someone that doesn't/hasn't I think your point is completely correct. It's better to not use ligatures for publication, such as in this scenario. (Now that I've said that, I better go check and see if I've made this mistake due to just not thinking about it.... hmm.....)
carrja99 2y ago
A lot of folks configure their editor to render |> as a rotated triangle.
- Kamq 2y ago
  Sure, but putting it in a code sample is similar to putting opening/closing quotes in a code sample instead of " It makes it harder for people to copy and paste and play with.
  - dpatterbee 2y ago
    If you copy and paste that triangle you will get "|>".
mise_en_place 2y ago
100% agree, it's just jarring to anyone who's developed in Elixir before. It's just like dquote characters on MacOS ("smart quotes")
LorenzoGood 2y ago
I, as a new elixir user, was personally confused by this exact thing as well.

conradfr 2y ago

Quite interesting, except the fork aspect.

Moosieus 2y ago
Yeah, it’s a non-starter in a lot of ways. I’m working on an alternate implementation that uses fragments, enabled by ongoing updates to ParadeDB’s syntax. Gonna explore all options and see where the cards land.

latch 2y ago

I don't understand why people use Ecto (or ActiveRecord, or...) Back in the day, I'm pretty sure we were using Hibernate and friend because our software was shipped and we wanted it to work with whatever database the client was using. But for a hosted software, what's the point? Not having to know SQL or details about PostgreSQL / the underlying DB ? Apps should be using SQL directly, and for cases where you need dynamic SQL (like, you're where clause is different based on some query string parameters), you can have a low-level query builder (1) (1) I'm not affiliated with it and have never used it, but a good search came up with https://github.com/robconery/moebius https://github.com/robconery/moebius which, at least from the readme, is roughly what I'm talking about.

isodev 2y ago
> we wanted it to work with whatever database While this is an option, making an app work with different database backends is not usually why one picks a library like Ecto or ActiveRecord. You see, these frameworks bring a tone of facilities for working with data, adding common solutions so we don't have re-invent the wheel every time. Everything from sanitation of user inputs and query building to creating update statements, validating input data and generating validation messages. Ecto is smart enough to even offer facilities for creating and handling forms based on the provided schema of the data. It also works with virtual data (not backed by a database) and all that can happen in Elixir, the same language as the rest of the application. For me, this has the advantage that I can focus on building what my app is supposed to do. Of course, if you really want or need to, there is always an option to submit your "direct SQL" query, and you will be free to write all the boilerplate needed to handle that.
prophesi 2y ago
Ecto isn't an ORM like ActiveRecord or Hibernate. It's a DSL for building queries and you can always drop down into raw SQL if you'd like. Moebius looks like Ecto except now you no longer have schema validations and the like.
- sph 2y ago
  Yes. It's more of a data mapper and composable DSL over SQL than an "object-relational mapper", since indeed there are no stateful objects in Elixir.

dugmartin 2y ago

If you are looking for a great blog post describing how to build a native full text search engine in Elixir that you can drop into your app: https://culttt.com/2023/03/22/building-a-full-text-search-engine-in-elixir https://culttt.com/2023/03/22/building-a-full-text-search-en... (I’ve no affiliation with the author)

skybrian 2y ago

Context: Ecto [1] seems to be a database layer for Elixir and Elixir is a programming language for Erlang’s virtual machine. [1] https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto