4 ms·

Hope you guys like this and use it! If it's hard to use, tell me. My goal is to make this simple to run anywhere. Part of that will be to use Distillery to crea

Hope you guys like this and use it! If it's hard to use, tell me. My goal is to make this simple to run anywhere. Part of that will be to use Distillery to create compiled executables. I rewrote this after an initial version that used Redis as a queue to process. Asking end users to install Redis was a step too far I think, so I stepped back and thought about how to solve this without external dependencies. I ended up with a simple Elixir and Erlang queue implementation. It works much simpler, just as fast, and no freaky deps. GenServer starts up a worker for each crawler, and it schedules work by itself. It's really strange to use a language so complete, it kind of feels like cheating. Programming in easy-mode. Pull requests very welcome to the crawler folder, it's super easy to write a crawler for your favorite site! https://github.com/sergiotapia/magnetissimo/tree/master/lib/crawler https://github.com/sergiotapia/magnetissimo/tree/master/lib/...

memla 10y ago
If the goal is to make it simple to run, why not support SQLite?
- brightball 10y ago
  Seems like the goal is avoiding dependencies and he's got 3 built in database choices for that already. DETS or Mnesia would probably work well for this.
  - sergiotapia 10y ago
    Thank you, I'll take a look at what those are. If there's a way to remove Postgres from the equation, I'm all for it.
    - macintux 10y ago
      DETS (and ETS, the in-memory equivalent) are quite simple for basic K/V-ish storage needs. If you need a relational database, Mnesia should do the trick.
    - yellowapple 10y ago
      Mnesia is probably what you want in that case, since it turns (D)ETS into a full-fledged database rather than a simple key-value store (think SQL v. Redis). Both are part of Erlang/OTP, though, so you get them for free just by using Erlang/Elixir/LFE/whatever. There's a hex.pm package called "Amnesia" that wraps Mnesia in a more Elixiry format.
      - sergiotapia 10y ago
        That sounds fantastic. I'm all for trimming the fat if we don't need it. I'll research this, thanks for the suggestion.
        elcritch 10y ago
        If you're using Phoenix or Ecto, look into ecto_mnesia (https://github.com/Nebo15/ecto_mnesia https://github.com/Nebo15/ecto_mnesia). It wraps up mnesia into a nice ecto wrapper. It's limited on what your primary keys can be.
gyrgtyn 10y ago
you could make it a nerves project too, maybe after you get rid of some of those database dependencies. It'll make you a ~100Mb linux distro that will boot straight into your app in ~10 seconds on a raspberri pi. http://nerves-project.org/ http://nerves-project.org/
- sauvage255 10y ago
  very useful project, thanks for sharing this!

memla 10y ago

If the goal is to make it simple to run, why not support SQLite?

brightball 10y ago
Seems like the goal is avoiding dependencies and he's got 3 built in database choices for that already. DETS or Mnesia would probably work well for this.
- sergiotapia 10y ago
  Thank you, I'll take a look at what those are. If there's a way to remove Postgres from the equation, I'm all for it.
  - macintux 10y ago
    DETS (and ETS, the in-memory equivalent) are quite simple for basic K/V-ish storage needs. If you need a relational database, Mnesia should do the trick.
  - yellowapple 10y ago
    Mnesia is probably what you want in that case, since it turns (D)ETS into a full-fledged database rather than a simple key-value store (think SQL v. Redis). Both are part of Erlang/OTP, though, so you get them for free just by using Erlang/Elixir/LFE/whatever. There's a hex.pm package called "Amnesia" that wraps Mnesia in a more Elixiry format.
    - sergiotapia 10y ago
      That sounds fantastic. I'm all for trimming the fat if we don't need it. I'll research this, thanks for the suggestion.
      - elcritch 10y ago
        If you're using Phoenix or Ecto, look into ecto_mnesia (https://github.com/Nebo15/ecto_mnesia https://github.com/Nebo15/ecto_mnesia). It wraps up mnesia into a nice ecto wrapper. It's limited on what your primary keys can be.

gyrgtyn 10y ago

you could make it a nerves project too, maybe after you get rid of some of those database dependencies. It'll make you a ~100Mb linux distro that will boot straight into your app in ~10 seconds on a raspberri pi. http://nerves-project.org/ http://nerves-project.org/

sauvage255 10y ago
very useful project, thanks for sharing this!