6 ms·
I'm building a podcast discovery app and I find myself being de-facto pulled towards modularity. It's because my feed checker is in Elixir, my site is WordPress
by firasd 10y ago
I'm building a podcast discovery app and I find myself being de-facto pulled towards modularity. It's because my feed checker is in Elixir, my site is WordPress-based, and I communicate between them using the WP API, and I'm using Google Cloud SQL, and Elasticsearch on its own virtual machine...
The thing is though, the Elixir feed checker has its own database table that tracks whether it's seen an episode in a feed. And when there's a new episode it sends an API call to WP to insert the new post. The problem is that sometimes the API calls fail! Now what? I'll need to build logging, re-try etc. So I'm thinking of making the feed checker 'stateless' and only using WP with a lot of query caching as the holder of 'state' information about whether an episode has been seen before.
To sum up my experience so far, there's something nice about being able to use the right tech for each task, and separating resources for each service, but the complexity--keeping track of whether a task completed properly--definitely increases.
- iagooar 10y agoCare sharing your progress with the podcast discovery app? I'm a cofounder of Podigee which is a podcast hosting service. Maybe we can exchange know-how, find some synergies or even join forces on certain topics. Feel free to drop me a line at mati@podigee.com
- firasd 10y agoSure, emailed. Also added a screenshot/twitter to my profile in case anyone else is interested.
- asimuvPR 10y agoI think your problem might be not expanding wordpress. PHP will gladly do the above through wordpress plugins and cron jobs. I built something similar and ended up going that way. The system is still running to this day. Sure, its not hip but gets the job done with miniml fuss and makes money.
- firasd 10y agoYou are right that PHP can do the feed checking part but I wanted to use something with easy async/concurrency out of the box (and wanted to learn Elixir instead of using Node.) One hard tech limit is that with 50k podcasts, 4million+ episodes, search definitely doesn't work well. Not just WP, but SQL itself. Hence Elasticsearch. I also plan to work on recommendations, etc. so will need probably to be exporting SQL data into other systems anyway for making the "people who liked this also liked this" kinda things. Also I kinda lied about using the WP API--that's how I built the system initially (and will switch to it moving forward), but to import the first few million posts from the content of the feeds, I just used wp_insert_post against the DB of new entries that Elixir fetched (I posted the code I used here: http://wordpress.stackexchange.com/a/233786/30906 http://wordpress.stackexchange.com/a/233786/30906). I also plan to write the whole front-end in React (including server side rendering) so will have to figure out how to get that done. Would probably use the WP-API with a Node.js app in front of it, will look into hypernova from AirBNB. So probably more usage of WP API accessed by another service...
- asimuvPR 10y agoI hope you are not doing all of this alone. I'd try and keep things as simple as possible within a monolith and then improve as needs increase. Good luck :)
- collyw 10y agoI generally write what I consider monolithic Django apps. I would add in Haystack (a search module for Django) and configure it to use Elasticsearch to overcome the problems you describe. It doesn't sound like microservices are needed, just adding in the appropriate tech for the job.
- jessaustin 10y agocron jobs Once these are doing anything other than rotating log files, can the system really be considered monolithic?
- asimuvPR 10y ago