12 ms·

Palantir acquires Kimono

171 points by davidbarker 11y ago

MeGer 11y ago
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but ParseHub is a more powerful and flexible web scraper. https://www.parsehub.com/ https://www.parsehub.com/
minimaxir 11y ago
As someone who used Kimono a few times before the closure...this announcement does not surprise me at all. The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono. Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.
- w1ntermute 11y ago
  Another one for http://ourincrediblejourney.tumblr.com/ http://ourincrediblejourney.tumblr.com/
- michaelmior 11y ago
  "too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?
  - minimaxir 11y ago
    Laziness is a secret form of technical debt. For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
    - michaelmior 11y ago
      While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).
      - minimaxir 11y ago
        See x1024's comment above. (best case scenario is that it's not significantly faster)
  - x1024 11y ago
    It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity: An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does. What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable. The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.
    - trhway 11y ago
      > "non-programmers can do it. No code required, etc. etc." translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
    - derefr 11y ago
      We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems." You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
      - x1024 11y ago
        I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.
- mrdrozdov 11y ago
  Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-intel-of-data-for-artificial-intelligence/ http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-i...
  - minimaxir 11y ago
    DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues. I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
    - mistermann 11y ago
      Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is: http://www.stockhouse.com/companies/bullboard/t.y/yellow-media-limited http://www.stockhouse.com/companies/bullboard/t.y/yellow-med... Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
      - dwynings 11y ago
        Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/ http://www.diffbot.com/products/crawlbot/ If you'd like me to set you up with a demo, feel free to email me: dru@diffbot.com
    - dwynings 11y ago
      Hi Max, All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3A%2F%2Fnews.ycombinator.com%2Fitem%3Fid%3D11105652 http://www.diffbot.com/testdrive/?api=discussion&url=https%3... If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/ http://www.diffbot.com/products/custom/
deleted 11y ago
[deleted]
svarrall 11y ago
Disappointing. Was my goto service as it was a joy to use and always seemed to work. Wish they would follow the Parse lead and release something open source. Desktop doesn't really cut it for API use...
marvel_boy 11y ago
Man, this is disappointing. No way to trust any cloud offering, more sooner than later the service is discontinued.
- davidbarker 11y ago
  I was just thinking the same. I rely on Kimono for a few of my sites, so I'm looking around at what I could use instead. The first I thought of is http://import.io/ http://import.io/, http://cloudscrape.com http://cloudscrape.com, or https://phantomjscloud.com/site/index.html https://phantomjscloud.com/site/index.html, but I fear the same might happen to whichever I pick. Does anyone have any suggestions on how to run something similar on my own server?
  - rurai 11y ago
    You can use portia[1]. You can try out a hosted version at scrapinghub[2] or you can host your own. It's a web based spider creator like kimonolabs but the spiders it creates are just scrapy spiders. If you need to do something that the UI can't handle it's not a problem, you can extend it with a little python. [1] https://github.com/scrapinghub/portia https://github.com/scrapinghub/portia [2] https://scrapinghub.com https://scrapinghub.com
  - tn13 11y ago
    Download the Kimono desktop tool for Mac and unzip it. It is nothing but a nodejs based web application.
- bttf 11y ago
  Perhaps a viable alternative to using cloud services would be roll-your-own OSS packages
  - deleted 11y ago
    [deleted]
- shift8 11y ago
  I agree. I actually referred people to Kimono. I'm so sorry I did. This is unacceptable behavior in this day and age. I mean seriously? Servers are cheap. Keep the thing up for existing users. Don't add new features. Don't invest any more time in the thing. But don't pull the rug from under people's feet. That's just bad form. If this is who Palantir acquires or acquihires, then they have some really bad judgement too. It reflects poorly on Palantir to not care about commitment and responsibility.
fapjacks 11y ago
"access to the world’s most important data problems" I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.
- trhway 11y ago
  > I can't imagine they're not pointing those tools at everyday Americans. http://www.dhs.gov/state-and-major-urban-area-fusion-centers http://www.dhs.gov/state-and-major-urban-area-fusion-centers Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
- jsprogrammer 11y ago
  Yeah, I'm not sure how they can drop such an unqualified statement like that. Who says Palantir has the world's most important data problems? Could someone give an example? Why is access to these most important problems limited to Palantir, or otherwise not accessible to most people (as implied by this shutdown notice)?
drglitch 11y ago
"Why? As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016" In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious. On the other hand, congrats to Kimono team!
- spesad 11y ago
  ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but I found ParseHub to be a more powerful and flexible web scraper.
- asdfologist 11y ago
  Well, if you want the real answer: Why? Money.
- code4tee 11y ago
  Sort of plays right into the whole argument being made over on another thread (https://news.ycombinator.com/item?id=11105198 https://news.ycombinator.com/item?id=11105198) that companies shouldn't rely too heavily on proprietary tools to solve their data challenges. Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
  - dkarapetyan 11y ago
    There is no mystery algorithm. It's just cheap college labor working obscene hours to pump out "data analysis" scripts. The sad thing is that it actually works. The places that hire palantir are so mired in bureaucracy and process that whatever palantir builds seems to be better. Of course, just like SAP and Accenture you're now kinda locked in with support contracts. Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.
gingerlime 11y ago
Sad to see Kimono close down. I enjoyed using it for a small project, and was waiting to see a paid version with private APIs, but it was always "coming soon"... I don't quite get why startups are not trying to generate revenue from day one.
jorgecurio 11y ago
I wonder what it's going to look like for import.io as well, their business model is identical to kimono it seems. edit: why the downvotes seriously.
- orliesaurus 11y ago
  Here's an upvote, because I was thinking exactly the same as someone who has used most of these "scraping" tools.
p4wnc6 11y ago
Palantir is one of those organizations that makes me throw up a little in the back of my throat whenever I hear about them. They are sort of like the Nickelback of companies.
- orng 11y ago
  I'm no fan of Nickleback's but I don't think they deserve to be compared to Palantir. If I was a Kimono dev I would certainly be leaving now.
  - smegel 11y ago
    Is it because what they do (helping governments snoop on social media) or are they actually a bad place to work?
    - orng 11y ago
      For morality reasons I would never be comfortable working there. The workplace could be heaven on earth and pay really well but there are simply things that are more important.
      - p4wnc6 11y ago
        The company culture is quite shitty as well.
mikaylutlin546 11y ago
Man, this is disappointing.
austinhutch 11y ago
I loved Kimono! It had the best interface and most flexibility for this type of service. A bittersweet congratulations to the team.
misterrobot 11y ago
The worst part about Palantir is their ability to masquerade as this hip startup, as though working for them is as innocent as working for Imgur or Twitter or whatever. They recruit the hell out of MIT students, and it sucks seeing my friends interview for this incredibly shitty company that has openly said it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?). I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.
- whorleater 11y ago
  Interning for Palantir is a pretty big resume boost for college students, and they have one of the highest paying internships in the valley. There's some serious benefits to going to Palantir.
  - misterrobot 11y ago
    If you have the ability to intern at Palantir you have the ability to intern at any number of more responsible companies. Palantir has a bigger booth at the career fair, but I'd bet working at Apple or Uber is just as much of a resume boost at much less of a societal cost. (source: am a college student)
    - whorleater 11y ago
      Interning at Apple or Uber has much more stiff competition though. The barrier to entry for Palantir is remarkably low considering their pay, it's not surprising that college students apply for Palantir. (Source: am college senior)
      - ZanyProgrammer 11y ago
        Will they hire you if you smoke pot? I hear college students sometimes do that.
kumarski 11y ago
Relevant Scraping/Crawling Lead Mining Tools: http://grabby.io http://grabby.io http://fullcontact.com http://fullcontact.com http://emailhunter.co http://emailhunter.co http://clearbit.com http://clearbit.com http://toofr.com http://toofr.com http://import.io http://import.io http://kimonolabs.com http://kimonolabs.com http://apifier.com http://apifier.com (favs) http://elink.club http://elink.club http://www.eliteproxyswitcher.com/ http://www.eliteproxyswitcher.com/ - ;)
- outdooricon 11y ago
  Was really hoping someone would share a comment like this, thank you. I feel like I just got completely screwed by Kimono...
- dwynings 11y ago
  http://www.diffbot.com http://www.diffbot.com
- henrik1409 11y ago
  http://cloudscrape.com http://cloudscrape.com :)
- ultimatejman 11y ago
  i have found import.io to be pretty good
  - Harshita 11y ago
    Have you tried pulling out anything from LinkedIn there?
- ddebernardy 11y ago
  Three more: https://github.com/scrapinghub/portia/ https://github.com/scrapinghub/portia/ (our open source equivalent of Kimonolabs) http://scrapy.org http://scrapy.org (allows to scrape more complex sites; also OSS) http://scrapinghub.com http://scrapinghub.com (for a cloud-based platform and a smart proxy rotator) (Disclaimer: working there.)
tn13 11y ago
Two months is a short time but I had sort of expected this given how bad their service of performing last few days. That is why I had started moving to other services.
- tobltobs 11y ago
  Its only two weeks, the equivalent of a GFY to their ex-customers.
nat 11y ago
As usual with acquisitions, they have stripped any information about what their product actually is (was?) from their site.
tn13 11y ago
To all those badmouthing Palantir. I think they are just a hogwash. They are just pretending to be something that Fox Maulder would investigate I dont think they are. They are just helping the government spend more money on problems that are not worth solving and create new problems for citizens. The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.
yueq 11y ago
This is just an acquihire.
tsergiu 11y ago
Full disclosure: I'm one of the co-founders of ParseHub. For folks looking for an alternative, check out http://www.parsehub.com http://www.parsehub.com ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns). Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.
- jorgecurio 11y ago
  This is a good list of alternatives: https://www.quora.com/Which-are-some-of-the-best-web-data-scraping-tools https://www.quora.com/Which-are-some-of-the-best-web-data-sc...
- nefitty 11y ago
  Awesome. Thanks for sharing your service. My current project has been faltering because of my ignorance of these types of tools. Yours looks like it's going to make a huge difference for me!
- dburn 11y ago
  Parsehub is easily the best hosted craping service on the market. We investigated everybody from kimono to import.io and if you want it done right they are the only option. Furthermore the guarantee of an open source version should they ever decide to close is extremely reassuring.
sergiotapia 11y ago
It's gross they're giving their customers 2 weeks to "GTFO".
oneloop 11y ago
We can conclude from this that they weren't doing very well, right? I mean, you don't accept an acquihire if you're doing great.
- orliesaurus 11y ago
  Its a two edged sword, sometimes you're doing great - but the fact they're shutting down means they're not doing great but have a team of experts in parsing that will benefit Palantir since they're likely into analysing surface and deep web data and just data in general :) basically its like hiring a bunch of regex gods because you have regex problems on large scale
nreece 11y ago
Disappointing to see just 2 weeks notice for their existing users. * shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com https://feedity.com Try it out and see if it works for your needs.
robertp 11y ago
ohhhhh man that stinks. I love Kimono and have been using for 6+ months. Hopefully it works out okay for the Kimono team. There is not a download function on the desktop page, what is the direct download link? https://www.kimonolabs.com/desktop https://www.kimonolabs.com/desktop
jpatokal 11y ago
Evaluated Kimono for a tool I was building, decided to roll my own with Scrapy instead. So happy right now that I did...
rmac 11y ago
Kimono had a beautiful chrome extension which allowed me to easily extract things quickly without needing to write any selectors / scraping. Much love to this team; best of luck!
danso 11y ago
From the FAQ: https://www.kimonolabs.com/faq#faq-303 https://www.kimonolabs.com/faq#faq-303 > Will Palantir (or anyone else) have access to my personal data? No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016. Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.
equivalentideas 11y ago
You can host scrapers you need to migrate on morph.io ( https://morph.io https://morph.io ) a free platform for hosting and running scrapers, storing your data and providing access to it in a range of formats (JSON, CSV, SQLite). It’s free, open source, supports a bunch of languages and is run by an Australian non-profit, where I happen to work ;-). Here's a tutorial all about running a scraper on morph.io https://www.openaustraliafoundation.org.au/2015/10/13/ruby-web-scraping-tutorial-on-morph-io-part-1/ https://www.openaustraliafoundation.org.au/2015/10/13/ruby-w...
foresight 11y ago
I'm always disappointed when I read of Palantir acquihires. Technically talented employees receive a pittance and are sold into the Palantir machine, a morally unscrupulous high-tech consulting shop that takes in talented engineers and churns out one-off software black boxes designed to lock in customers and extract a long-term tax.
tn13 11y ago
Their "desktop tool" is complete rubbish. I have to manually click on each source to crawl it and then manually click a few buttons to save the response. Thanks for nothing guys!
uberneo 11y ago
Looks like we cannot schedule an API in the desktop version
uberneo 11y ago
few other options using NodeJs - http://blog.webkid.io/nodejs-scraping-libraries/ http://blog.webkid.io/nodejs-scraping-libraries/
deleted 11y ago
[deleted]
solveforall 11y ago
I had a hard time using this product. I was trying to use another developer's APIs but it didn't seem possible. The one positive thing about the announcement, from the FAQ: Will Palantir (or anyone else) have access to my personal data? No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016. https://www.kimonolabs.com/faq https://www.kimonolabs.com/faq
shaneaevans 11y ago
A similar thing happened to Needlebase, another visual scraping tool, which was bought by google in 2012 and shut down. Before that there was dapper.. Shameless plug: At Scrapinghub, we're committed to open source and Portia (http://scrapinghub.com/portia/ http://scrapinghub.com/portia/) is the only open source alternative to Kimono. We'll be providing an easy migration path for Kimono users.
jd2025 11y ago
Anyone offering a GUI solution that can export to an RSS feed like Kimono did? I know RSS is old but I have no clue how to display the other formats in wordpress. Thanks!
tn13 11y ago
These people should be totally ashamed for the tool they have given. It is just nodejs wrapped inside a desktop app. They could have very well made it open sourced and offered to community.

171 points by davidbarker 11y ago

MeGer 11y ago

ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but ParseHub is a more powerful and flexible web scraper. https://www.parsehub.com/ https://www.parsehub.com/

minimaxir 11y ago

As someone who used Kimono a few times before the closure...this announcement does not surprise me at all. The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono. Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.

w1ntermute 11y ago
Another one for http://ourincrediblejourney.tumblr.com/ http://ourincrediblejourney.tumblr.com/
michaelmior 11y ago
"too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?
- minimaxir 11y ago
  Laziness is a secret form of technical debt. For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
  - michaelmior 11y ago
    While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).
    - minimaxir 11y ago
      See x1024's comment above. (best case scenario is that it's not significantly faster)
- x1024 11y ago
  It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity: An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does. What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable. The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.
  - trhway 11y ago
    > "non-programmers can do it. No code required, etc. etc." translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
  - derefr 11y ago
    We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems." You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
    - x1024 11y ago
      I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.
mrdrozdov 11y ago
Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-intel-of-data-for-artificial-intelligence/ http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-i...
- minimaxir 11y ago
  DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues. I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
  - mistermann 11y ago
    Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is: http://www.stockhouse.com/companies/bullboard/t.y/yellow-media-limited http://www.stockhouse.com/companies/bullboard/t.y/yellow-med... Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
    - dwynings 11y ago
      Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/ http://www.diffbot.com/products/crawlbot/ If you'd like me to set you up with a demo, feel free to email me: dru@diffbot.com
  - dwynings 11y ago
    Hi Max, All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3A%2F%2Fnews.ycombinator.com%2Fitem%3Fid%3D11105652 http://www.diffbot.com/testdrive/?api=discussion&url=https%3... If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/ http://www.diffbot.com/products/custom/

deleted 11y ago

[deleted]

svarrall 11y ago

Disappointing. Was my goto service as it was a joy to use and always seemed to work. Wish they would follow the Parse lead and release something open source. Desktop doesn't really cut it for API use...

marvel_boy 11y ago

Man, this is disappointing. No way to trust any cloud offering, more sooner than later the service is discontinued.

davidbarker 11y ago
I was just thinking the same. I rely on Kimono for a few of my sites, so I'm looking around at what I could use instead. The first I thought of is http://import.io/ http://import.io/, http://cloudscrape.com http://cloudscrape.com, or https://phantomjscloud.com/site/index.html https://phantomjscloud.com/site/index.html, but I fear the same might happen to whichever I pick. Does anyone have any suggestions on how to run something similar on my own server?
- rurai 11y ago
  You can use portia[1]. You can try out a hosted version at scrapinghub[2] or you can host your own. It's a web based spider creator like kimonolabs but the spiders it creates are just scrapy spiders. If you need to do something that the UI can't handle it's not a problem, you can extend it with a little python. [1] https://github.com/scrapinghub/portia https://github.com/scrapinghub/portia [2] https://scrapinghub.com https://scrapinghub.com
- tn13 11y ago
  Download the Kimono desktop tool for Mac and unzip it. It is nothing but a nodejs based web application.
bttf 11y ago
Perhaps a viable alternative to using cloud services would be roll-your-own OSS packages
- deleted 11y ago
  [deleted]
shift8 11y ago
I agree. I actually referred people to Kimono. I'm so sorry I did. This is unacceptable behavior in this day and age. I mean seriously? Servers are cheap. Keep the thing up for existing users. Don't add new features. Don't invest any more time in the thing. But don't pull the rug from under people's feet. That's just bad form. If this is who Palantir acquires or acquihires, then they have some really bad judgement too. It reflects poorly on Palantir to not care about commitment and responsibility.

fapjacks 11y ago

"access to the world’s most important data problems" I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.

trhway 11y ago
> I can't imagine they're not pointing those tools at everyday Americans. http://www.dhs.gov/state-and-major-urban-area-fusion-centers http://www.dhs.gov/state-and-major-urban-area-fusion-centers Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
jsprogrammer 11y ago
Yeah, I'm not sure how they can drop such an unqualified statement like that. Who says Palantir has the world's most important data problems? Could someone give an example? Why is access to these most important problems limited to Palantir, or otherwise not accessible to most people (as implied by this shutdown notice)?

drglitch 11y ago

"Why? As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016" In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious. On the other hand, congrats to Kimono team!

spesad 11y ago
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but I found ParseHub to be a more powerful and flexible web scraper.
asdfologist 11y ago
Well, if you want the real answer: Why? Money.
code4tee 11y ago
Sort of plays right into the whole argument being made over on another thread (https://news.ycombinator.com/item?id=11105198 https://news.ycombinator.com/item?id=11105198) that companies shouldn't rely too heavily on proprietary tools to solve their data challenges. Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
- dkarapetyan 11y ago
  There is no mystery algorithm. It's just cheap college labor working obscene hours to pump out "data analysis" scripts. The sad thing is that it actually works. The places that hire palantir are so mired in bureaucracy and process that whatever palantir builds seems to be better. Of course, just like SAP and Accenture you're now kinda locked in with support contracts. Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.

gingerlime 11y ago

Sad to see Kimono close down. I enjoyed using it for a small project, and was waiting to see a paid version with private APIs, but it was always "coming soon"... I don't quite get why startups are not trying to generate revenue from day one.

jorgecurio 11y ago

I wonder what it's going to look like for import.io as well, their business model is identical to kimono it seems. edit: why the downvotes seriously.

orliesaurus 11y ago
Here's an upvote, because I was thinking exactly the same as someone who has used most of these "scraping" tools.

p4wnc6 11y ago

Palantir is one of those organizations that makes me throw up a little in the back of my throat whenever I hear about them. They are sort of like the Nickelback of companies.

orng 11y ago
I'm no fan of Nickleback's but I don't think they deserve to be compared to Palantir. If I was a Kimono dev I would certainly be leaving now.
- smegel 11y ago
  Is it because what they do (helping governments snoop on social media) or are they actually a bad place to work?
  - orng 11y ago
    For morality reasons I would never be comfortable working there. The workplace could be heaven on earth and pay really well but there are simply things that are more important.
    - p4wnc6 11y ago
      The company culture is quite shitty as well.

mikaylutlin546 11y ago

Man, this is disappointing.

austinhutch 11y ago

I loved Kimono! It had the best interface and most flexibility for this type of service. A bittersweet congratulations to the team.

misterrobot 11y ago

The worst part about Palantir is their ability to masquerade as this hip startup, as though working for them is as innocent as working for Imgur or Twitter or whatever. They recruit the hell out of MIT students, and it sucks seeing my friends interview for this incredibly shitty company that has openly said it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?). I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.

whorleater 11y ago
Interning for Palantir is a pretty big resume boost for college students, and they have one of the highest paying internships in the valley. There's some serious benefits to going to Palantir.
- misterrobot 11y ago
  If you have the ability to intern at Palantir you have the ability to intern at any number of more responsible companies. Palantir has a bigger booth at the career fair, but I'd bet working at Apple or Uber is just as much of a resume boost at much less of a societal cost. (source: am a college student)
  - whorleater 11y ago
    Interning at Apple or Uber has much more stiff competition though. The barrier to entry for Palantir is remarkably low considering their pay, it's not surprising that college students apply for Palantir. (Source: am college senior)
    - ZanyProgrammer 11y ago
      Will they hire you if you smoke pot? I hear college students sometimes do that.

kumarski 11y ago

Relevant Scraping/Crawling Lead Mining Tools: http://grabby.io http://grabby.io http://fullcontact.com http://fullcontact.com http://emailhunter.co http://emailhunter.co http://clearbit.com http://clearbit.com http://toofr.com http://toofr.com http://import.io http://import.io http://kimonolabs.com http://kimonolabs.com http://apifier.com http://apifier.com (favs) http://elink.club http://elink.club http://www.eliteproxyswitcher.com/ http://www.eliteproxyswitcher.com/ - ;)

outdooricon 11y ago
Was really hoping someone would share a comment like this, thank you. I feel like I just got completely screwed by Kimono...
dwynings 11y ago
http://www.diffbot.com http://www.diffbot.com
henrik1409 11y ago
http://cloudscrape.com http://cloudscrape.com :)
ultimatejman 11y ago
i have found import.io to be pretty good
- Harshita 11y ago
  Have you tried pulling out anything from LinkedIn there?
ddebernardy 11y ago
Three more: https://github.com/scrapinghub/portia/ https://github.com/scrapinghub/portia/ (our open source equivalent of Kimonolabs) http://scrapy.org http://scrapy.org (allows to scrape more complex sites; also OSS) http://scrapinghub.com http://scrapinghub.com (for a cloud-based platform and a smart proxy rotator) (Disclaimer: working there.)

tn13 11y ago

Two months is a short time but I had sort of expected this given how bad their service of performing last few days. That is why I had started moving to other services.

tobltobs 11y ago
Its only two weeks, the equivalent of a GFY to their ex-customers.

nat 11y ago

As usual with acquisitions, they have stripped any information about what their product actually is (was?) from their site.

tn13 11y ago

To all those badmouthing Palantir. I think they are just a hogwash. They are just pretending to be something that Fox Maulder would investigate I dont think they are. They are just helping the government spend more money on problems that are not worth solving and create new problems for citizens. The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.

yueq 11y ago

This is just an acquihire.

tsergiu 11y ago

Full disclosure: I'm one of the co-founders of ParseHub. For folks looking for an alternative, check out http://www.parsehub.com http://www.parsehub.com ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns). Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.

jorgecurio 11y ago
This is a good list of alternatives: https://www.quora.com/Which-are-some-of-the-best-web-data-scraping-tools https://www.quora.com/Which-are-some-of-the-best-web-data-sc...
nefitty 11y ago
Awesome. Thanks for sharing your service. My current project has been faltering because of my ignorance of these types of tools. Yours looks like it's going to make a huge difference for me!
dburn 11y ago
Parsehub is easily the best hosted craping service on the market. We investigated everybody from kimono to import.io and if you want it done right they are the only option. Furthermore the guarantee of an open source version should they ever decide to close is extremely reassuring.

sergiotapia 11y ago

It's gross they're giving their customers 2 weeks to "GTFO".

oneloop 11y ago

We can conclude from this that they weren't doing very well, right? I mean, you don't accept an acquihire if you're doing great.

orliesaurus 11y ago
Its a two edged sword, sometimes you're doing great - but the fact they're shutting down means they're not doing great but have a team of experts in parsing that will benefit Palantir since they're likely into analysing surface and deep web data and just data in general :) basically its like hiring a bunch of regex gods because you have regex problems on large scale

nreece 11y ago

Disappointing to see just 2 weeks notice for their existing users. * shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com https://feedity.com Try it out and see if it works for your needs.

robertp 11y ago

ohhhhh man that stinks. I love Kimono and have been using for 6+ months. Hopefully it works out okay for the Kimono team. There is not a download function on the desktop page, what is the direct download link? https://www.kimonolabs.com/desktop https://www.kimonolabs.com/desktop

jpatokal 11y ago

Evaluated Kimono for a tool I was building, decided to roll my own with Scrapy instead. So happy right now that I did...

rmac 11y ago

Kimono had a beautiful chrome extension which allowed me to easily extract things quickly without needing to write any selectors / scraping. Much love to this team; best of luck!

danso 11y ago

From the FAQ: https://www.kimonolabs.com/faq#faq-303 https://www.kimonolabs.com/faq#faq-303 > Will Palantir (or anyone else) have access to my personal data? No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016. Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.

equivalentideas 11y ago

You can host scrapers you need to migrate on morph.io ( https://morph.io https://morph.io ) a free platform for hosting and running scrapers, storing your data and providing access to it in a range of formats (JSON, CSV, SQLite). It’s free, open source, supports a bunch of languages and is run by an Australian non-profit, where I happen to work ;-). Here's a tutorial all about running a scraper on morph.io https://www.openaustraliafoundation.org.au/2015/10/13/ruby-web-scraping-tutorial-on-morph-io-part-1/ https://www.openaustraliafoundation.org.au/2015/10/13/ruby-w...

foresight 11y ago

I'm always disappointed when I read of Palantir acquihires. Technically talented employees receive a pittance and are sold into the Palantir machine, a morally unscrupulous high-tech consulting shop that takes in talented engineers and churns out one-off software black boxes designed to lock in customers and extract a long-term tax.

tn13 11y ago

Their "desktop tool" is complete rubbish. I have to manually click on each source to crawl it and then manually click a few buttons to save the response. Thanks for nothing guys!

uberneo 11y ago

Looks like we cannot schedule an API in the desktop version

uberneo 11y ago

few other options using NodeJs - http://blog.webkid.io/nodejs-scraping-libraries/ http://blog.webkid.io/nodejs-scraping-libraries/

deleted 11y ago

[deleted]

solveforall 11y ago

I had a hard time using this product. I was trying to use another developer's APIs but it didn't seem possible. The one positive thing about the announcement, from the FAQ: Will Palantir (or anyone else) have access to my personal data? No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016. https://www.kimonolabs.com/faq https://www.kimonolabs.com/faq

shaneaevans 11y ago

A similar thing happened to Needlebase, another visual scraping tool, which was bought by google in 2012 and shut down. Before that there was dapper.. Shameless plug: At Scrapinghub, we're committed to open source and Portia (http://scrapinghub.com/portia/ http://scrapinghub.com/portia/) is the only open source alternative to Kimono. We'll be providing an easy migration path for Kimono users.

jd2025 11y ago

Anyone offering a GUI solution that can export to an RSS feed like Kimono did? I know RSS is old but I have no clue how to display the other formats in wordpress. Thanks!

tn13 11y ago

These people should be totally ashamed for the tool they have given. It is just nodejs wrapped inside a desktop app. They could have very well made it open sourced and offered to community.