4 ms·
Some years ago, I worked for a big box retailer on their public-facing catalog api. It was a pretty exotic thing when it came out, as that data was all trade-se
by beat 7y ago
Some years ago, I worked for a big box retailer on their public-facing catalog api. It was a pretty exotic thing when it came out, as that data was all trade-secretive - which was dumb, because then they just had web scrapers brutalizing their front end to get the data that was public anyway.
So on the back end, the api collected data from many different sources within this very large company, and put it in nice api form. This was technically cached data, and could get stale, but really, most of it was sufficiently up-to-date most of the time.
Turned out the biggest customer for this data was the company itself. Lots of individual projects, needing some obscure piece of data or another, could get it from the api, rather than having to hunt for whatever department actually owned the data, and working out access arrangements on a piecemeal basis. I'm sure it saved countless thousands of hours of work, and greatly improved both quality and time to market for projects.
Looks like maybe IKEA could use something like that...
- RhodesianHunter 7y agoThat's funny, for a few years I worked for an organization that did the front end crawling / brutalizing of most of the major retailers for their data. Afaik most still do not intentionally provide any data publicly and many spend extravagant amounts on anti-crawling tech to prevent it, resulting in an expensive arms race over ostensibly public data.
- beat 7y agoYeah, it's just dumb, pseudo-paranoid thinking. Crawlers are going to get it anyway, and it's publicly available, so why not make it easier for everyone? And like I said, our biggest customer was our own company. It saved an incredible amount of hassle.
- marcosdumay 7y agoI always wondered what those companies think they'll gain by hiding the products they sell.
- gregmac 7y agoIf you put something on a public site it can be harvested, and there's really no stopping that. It's like video DRM: you can spend billions on the tech, but ultimately, if nothing else, anyone can still aim any camera at the screen and make a copy. There's only two reasons I can think of that companies would bother with this: 1) they are ignorant or delusional about it and think they can beat the crawler (which could be a third party making money by convincing them they can do it) 2) It forces the crawler to spend more money chasing their blocking I'm not sure what the benefit of 2 would be, though, unless you could make them spend so much money that their entire operation is unprofitable. Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly.
- leggomylibro 7y ago>Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly. Well if you replace 'scrapers' with 'advertisements', there are plenty of worthwhile OSS projects that might welcome help :) Making that sort of business unprofitable might not be a super realistic goal, though.
- Benjammer 7y agoThis is very reminiscent of the Bezos API memo at Amazon a while back...
- dtech 7y agoFor the interested: https://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf https://homepages.dcc.ufmg.br/~mtov/pmcc/modularization.pdf
- JetezLeLogin 7y agoAbsolutely - most bigger companies have silos and can make excellent use of things like this.