5 ms·

Advanced Image Manipulation and Data Extraction

103 points by mikeyanderson 9y ago

peckjon 9y ago
Author here! Ask me anything :-)
- deleted 9y ago
  [deleted]
- pwaai 9y ago
  What would be the advantage that Algorithmia offers vs building my own using my own training data? Is Algorithmia like a marketplace where I can list my own API? What are the response time like since it's serverless?
  - peckjon 9y ago
    With Algorithmia you can write algos in Python, R, Scala, Rust, Java, Ruby and JavaScript. And you can call those algos from any language at all. There’s solid dependency support, GPUs enabled, built-in versioning, automatic JSON APIs, etc. and the latency is about 15ms for warm responses. While you’d still want to train your model offline or elsewhere in your own preferred environment, this provides a DevOps-free environment for hosting your algo. You don’t have to worry about building a VM, manually scaling load, etc. And algos are easily chainable / pipeline-able across languages and developers. So your local Python script can call a remote R routine which loops over a Java function, with basically zero overhead. Algorithmia only charges per compute-second for use, not for hosting. And you can either make your API free, or charge a royalty per use. Refs: Creating an algo: https://algorithmia.com/developers/algorithm-development/languages/ https://algorithmia.com/developers/algorithm-development/lan... Calling an algo: https://algorithmia.com/developers/clients/ https://algorithmia.com/developers/clients/ Pricing: https://algorithmia.com/pricing https://algorithmia.com/pricing Platform overview: https://algorithmia.com/serverless-ai-layer https://algorithmia.com/serverless-ai-layer
    - pwaai 9y ago
      Thanks for the detailed answer....but I still don't get the why....but maybe because this is more for large teams? I'm struggling to understand why I would use this over Google Cloud or Microsoft AI. Am I misunderstanding the big picture here?
      - peckjon 9y ago
        There are great reasons to use GCloud, MS AI, and the many other offerings out there. The ideal solution will vary by use case. Some considerations might be: - serverless functions or full VM? With the former you pay by the second or minute, the latter is often by the hour or even month and many people use under 20% of the capacity they're paying for. - what languages do you need? Many full VMs can support anything, but most serverless function hosts support very few languages. - what frameworks do you need? Do you want to do a full install and dependency reconciliation, or have it done / managed for you? - interoperability, API chaining, and marketplace. Algorithmia's language-agnostic open marketplace is pretty unique: you can use grab any of 4500+ APIs right from your own algo, without ever leaving the environment. Happy to step through specific use-cases with you anytime! Just click the chatbox in the lower-right at Algorithmia.com
        pwaai 9y ago
        My comment is gonna sound harsh but I'm trying to understand what exactly the value is. How do the OCR api in the marketplace holds up against Google Cloud, which is considered the best in the industry? Do you see what I mean? All of the other things you've listed just wraps the core thing which is the actual algorithm. The serverless or having full VM doesn't cost much nor is it painful to setup imho. The language in the end does not matter since a developer could just make HTTP calls to the Google Cloud API. What good is having thousands of API when I just need a few that I know is the best in the industry? I'm curious to know if there are real world use cases here. Perhaps for larger companies, what you offer might be important. Thanks and I think that's the last of my questions!
        9y ago
tfmatt 9y ago
https://news.ycombinator.com/item?id=15965330 This was featured on HN a few days ago, it uses artifacts from the manufacturing process to extract data about which device took a photo. Is there any sort of algorithm on algorithmia at the moment that can give you this kind of information? Thank you, loved the post.
- peckjon 9y ago
  Thanks! Awesome idea. We don’t have such an algo yet but would love to bring one into our platform. The link isn’t working for me — do you know if it is based on https://pdfs.semanticscholar.org/1526/db85a39d246c83f86d346962ebc0030b7b58.pdf https://pdfs.semanticscholar.org/1526/db85a39d246c83f86d3469... ? If we can get the dataset and methodology, we (or a community member!) can implement it and make it available & extendable for all to use: https://algorithmia.com/developers/algorithm-development/ https://algorithmia.com/developers/algorithm-development/
  - tfmatt 9y ago
    https://gcn.com/articles/2017/12/19/smartphone-authentication-prnu.aspx https://gcn.com/articles/2017/12/19/smartphone-authenticatio... The application is specific in this instance but can obviously be generalized. Thanks for the info, I'll keep this in mind.
peterburkimsher 9y ago
SmartTextExtraction looks like exactly what I need! However, I have to be able to run offline. How can I download it and run the algorithm locally? Tesseract gave me pretty poor results, probably because I'm using photos of printed pages instead of flatbed scans.
- dan_quixote 9y ago
  I have never achieved reliable results with Tesseract regardless of image quality.
- mikeyanderson 9y ago
  We have the source code for that algo: https://algorithmia.com/algorithms/character_recognition/tesseract/edit https://algorithmia.com/algorithms/character_recognition/tes...
- newusertoday 9y ago
  did you tried pre processing the images before giving it to tesseract? i had very good results after pre processing the images.
hivacruz 9y ago
Finally registered to Hacker News just to say thanks for this. I wanted to use Google Cloud for so long but because I'm in Europe, I can't use it for non-business purpose. The algorithms I'm interested in seem to work pretty well so far for what I tested (text detection).
- mikeyanderson 9y ago
  Awesome. We're always available to help with anything you need. We have a chat box on every page and engineers ready to help. :)
enriquto 9y ago
There is some overlap with the IPOL journal : http://www.ipol.im/ http://www.ipol.im/ It is an image processing journal where the source code of the articles is peer-reviewed, and all the algorithms can be tested online.
- peckjon 9y ago
  Great reference — thanks! Hopefully we can bring some of these algos into our platform!

103 points by mikeyanderson 9y ago

peckjon 9y ago

Author here! Ask me anything :-)

deleted 9y ago
[deleted]
pwaai 9y ago
What would be the advantage that Algorithmia offers vs building my own using my own training data? Is Algorithmia like a marketplace where I can list my own API? What are the response time like since it's serverless?
- peckjon 9y ago
  With Algorithmia you can write algos in Python, R, Scala, Rust, Java, Ruby and JavaScript. And you can call those algos from any language at all. There’s solid dependency support, GPUs enabled, built-in versioning, automatic JSON APIs, etc. and the latency is about 15ms for warm responses. While you’d still want to train your model offline or elsewhere in your own preferred environment, this provides a DevOps-free environment for hosting your algo. You don’t have to worry about building a VM, manually scaling load, etc. And algos are easily chainable / pipeline-able across languages and developers. So your local Python script can call a remote R routine which loops over a Java function, with basically zero overhead. Algorithmia only charges per compute-second for use, not for hosting. And you can either make your API free, or charge a royalty per use. Refs: Creating an algo: https://algorithmia.com/developers/algorithm-development/languages/ https://algorithmia.com/developers/algorithm-development/lan... Calling an algo: https://algorithmia.com/developers/clients/ https://algorithmia.com/developers/clients/ Pricing: https://algorithmia.com/pricing https://algorithmia.com/pricing Platform overview: https://algorithmia.com/serverless-ai-layer https://algorithmia.com/serverless-ai-layer
  - pwaai 9y ago
    Thanks for the detailed answer....but I still don't get the why....but maybe because this is more for large teams? I'm struggling to understand why I would use this over Google Cloud or Microsoft AI. Am I misunderstanding the big picture here?
    - peckjon 9y ago
      There are great reasons to use GCloud, MS AI, and the many other offerings out there. The ideal solution will vary by use case. Some considerations might be: - serverless functions or full VM? With the former you pay by the second or minute, the latter is often by the hour or even month and many people use under 20% of the capacity they're paying for. - what languages do you need? Many full VMs can support anything, but most serverless function hosts support very few languages. - what frameworks do you need? Do you want to do a full install and dependency reconciliation, or have it done / managed for you? - interoperability, API chaining, and marketplace. Algorithmia's language-agnostic open marketplace is pretty unique: you can use grab any of 4500+ APIs right from your own algo, without ever leaving the environment. Happy to step through specific use-cases with you anytime! Just click the chatbox in the lower-right at Algorithmia.com
      - pwaai 9y ago
        My comment is gonna sound harsh but I'm trying to understand what exactly the value is. How do the OCR api in the marketplace holds up against Google Cloud, which is considered the best in the industry? Do you see what I mean? All of the other things you've listed just wraps the core thing which is the actual algorithm. The serverless or having full VM doesn't cost much nor is it painful to setup imho. The language in the end does not matter since a developer could just make HTTP calls to the Google Cloud API. What good is having thousands of API when I just need a few that I know is the best in the industry? I'm curious to know if there are real world use cases here. Perhaps for larger companies, what you offer might be important. Thanks and I think that's the last of my questions!
        9y ago

tfmatt 9y ago

https://news.ycombinator.com/item?id=15965330 This was featured on HN a few days ago, it uses artifacts from the manufacturing process to extract data about which device took a photo. Is there any sort of algorithm on algorithmia at the moment that can give you this kind of information? Thank you, loved the post.

peckjon 9y ago
Thanks! Awesome idea. We don’t have such an algo yet but would love to bring one into our platform. The link isn’t working for me — do you know if it is based on https://pdfs.semanticscholar.org/1526/db85a39d246c83f86d346962ebc0030b7b58.pdf https://pdfs.semanticscholar.org/1526/db85a39d246c83f86d3469... ? If we can get the dataset and methodology, we (or a community member!) can implement it and make it available & extendable for all to use: https://algorithmia.com/developers/algorithm-development/ https://algorithmia.com/developers/algorithm-development/
- tfmatt 9y ago
  https://gcn.com/articles/2017/12/19/smartphone-authentication-prnu.aspx https://gcn.com/articles/2017/12/19/smartphone-authenticatio... The application is specific in this instance but can obviously be generalized. Thanks for the info, I'll keep this in mind.

peterburkimsher 9y ago

SmartTextExtraction looks like exactly what I need! However, I have to be able to run offline. How can I download it and run the algorithm locally? Tesseract gave me pretty poor results, probably because I'm using photos of printed pages instead of flatbed scans.

dan_quixote 9y ago
I have never achieved reliable results with Tesseract regardless of image quality.
mikeyanderson 9y ago
We have the source code for that algo: https://algorithmia.com/algorithms/character_recognition/tesseract/edit https://algorithmia.com/algorithms/character_recognition/tes...
newusertoday 9y ago
did you tried pre processing the images before giving it to tesseract? i had very good results after pre processing the images.

hivacruz 9y ago

Finally registered to Hacker News just to say thanks for this. I wanted to use Google Cloud for so long but because I'm in Europe, I can't use it for non-business purpose. The algorithms I'm interested in seem to work pretty well so far for what I tested (text detection).

mikeyanderson 9y ago
Awesome. We're always available to help with anything you need. We have a chat box on every page and engineers ready to help. :)

enriquto 9y ago

There is some overlap with the IPOL journal : http://www.ipol.im/ http://www.ipol.im/ It is an image processing journal where the source code of the articles is peer-reviewed, and all the algorithms can be tested online.

peckjon 9y ago
Great reference — thanks! Hopefully we can bring some of these algos into our platform!