4 ms·
Couldn't you potentially push that aspect elsewhere, into a data storage solution like S3 or something (not sure of the cost efficiency, not a huge cloud guy my
by shortstuffsushi 11y ago
Couldn't you potentially push that aspect elsewhere, into a data storage solution like S3 or something (not sure of the cost efficiency, not a huge cloud guy myself). Then you'd just be utilizing the DO server for processing. It would have the added benefit of allowing you to scale your crawling horizontally if you wanted, as all your servers could have access to the same set of data.
- icebraining 11y agoThe traffic between datacenters would be significantly slower and more costly, though. Besides, S3 is not a replacement for a real drive.
- shortstuffsushi 11y agoIt would be slower than on disk for sure, so if that's a concern, it wouldn't be a good choice. If the poster is just storing off results, pushing it to S3, or even potentially a document database could be a logical choice. They said they're just using it for occasionally re-parsing at a later date, so immediacy doesn't seem like a huge concern. Just a thought, at any rate.
- deleted 11y ago[deleted]
- 3pt14159 11y agoYeah I could, and I currently do - but I don't like to. I want the data on the same network for lots of reasons and I also don't like maintaining two sets of deployment systems. I'd prefer to just use Cloud66 + Digital Ocean and focus on writing code, not setting up and maintaining servers.
- shortstuffsushi 11y agoOh yeah, I totally understand that, was just offering an alternative as a non-ideal workaround. I have the same set up for a service I offer, primarily because (afaik) it's hard to beat S3 pricing.