4 ms·
Ask HN: What do you use to move (copy) data between systems?
Do you centralize your data? Do you run an iPaaS of some kind? Do you like it?
Context: I'm currently in the middle of a project where I'm hooking up an ERP system (SalesPad) to an eCommerce website (WooCommerce) and I've found I need to move many different kinds of data between them (Inventory/Orders/CustomPricing/etc), and I haven't found a good tool so I started building one.
Has anyone else run into this? How did you solve it? Did you like your solution?
- linsomniac 6y agoMany of these sorts of things we will push to an S3 bucket, and have other parts of the system pull from that bucket. We also use "rsync" with SSH keys to copy some data. Some of our vendors provide data to us via FTP, similarly it could be provided via HTTP Maybe Zapier can help?
- Jugurtha 6y agoCan you configure database replication so your database handles the transfer to the target system, and then make the ecommerce site accessible for orders? The two databases will be kept in sync.
- toomuchtodo 6y agoHave you checked out http://n8n.io http://n8n.io? Sounds like you’re just pushing JSON blobs around on a bus.
- sitkack 6y agorsync, rclone really depends on what formats the data is in, how it will be used, how long it needs to last, how long the code that reads and writes the data will last, how much out of band information vs inband information will be sent along with the data. You talked about movement, movement is the last of your worries, formats and provenance should rank higher on your list of concerns. My gotos are sqlite, xml and avro. Try to keep things under 2GB so you don't run into lame data transfer issues.
- nailer 6y agorsync. Because it's efficient and well understood and emphasises accuracy.
- alexchantavy 6y agoTo add to the rsync answers, I think robocopy in a Windows environment is great.
- eb0la 6y agoTake a look at kafka ;-) Just put the file path as the key and the content as the message value. You get TTL/size garbage collection for free: no need to get/develop a sophisticated client/server.
- beeve 6y agoLots of data integration software is available. In the open source world, Apache Nifi and Apache Camel are two examples.