5 ms·

There's a conflict in value proposition here, which is more drastic than the discovery issue. That might help you better contextualize the failure, and avoid si

There's a conflict in value proposition here, which is more drastic than the discovery issue. That might help you better contextualize the failure, and avoid similar risks in the future. Namely, GPT is now a multimodal trained llm capable of doing OCR over PDFs and images. Given the accessibility of that feature, we expect that users do not try to discover OCR plugins anymore- they feel no need. Out of curiosity, what was the contingency plan in the case that OpenAI did this? What rationales did you use to estimate the likelihood and severity of that risk? Were there good reasons to discount that risk?

Areibman 3y ago
We knew the time would come, but we built ChatOCR in a week. If we overthought the time horizon problem, we'd have 0 users and $0. But also, GPT-4-Vision is multi-modal but does not specifically use OCR. Our tool is used mostly to extract text from documents and load it into context, and we still saw growth after OpenAI built this feature into ChatGPT.
- nextworddev 3y ago
  $20k for a week’s work seems more than good, no?
  - sangnoir 3y ago
    Whose week's work? One person, a small team (<5), or a large organization? At some point on that axis, the RoI becomes negative.
    - guiambros 3y ago
      You can't get a large organization to move fast and ship something in a week. So this was either a solo developer en devour, or at max low single digits.
- barfbagginus 3y ago
  Thank you for sharing your risk thinking here. I like that you are risk hungry, not prone to hesitation, and you achieved revenue on a shoe string budget. I'm curious about the idea that your OCR-generated context might give better recall than whatever GPT uses to parse PDFs into context. That would support the idea that it is discoverability, and not feature parity, that is killing the product. Can we discuss benchmarks that show conclusively that your solution does better on QA accuracy with respect to a PDF, as compared to GPT4's own PDF mode? Also I'm curious about the trajectory of active users after the feature dropped from OAI. Was there an inflection point where new active users per month went negative? How long did it take before that inflection happened? Do you still have enough users to keep the service up and gathering revenue, or do you plan to mothball? Do you have any plans to pivot the offering to other markets?
yaj54 3y ago
What resources do you suggest for learning how to do this kind of risk analysis well?
- vasco 3y ago
  A crystal ball. Most of the times you should do what they did, build a first version in a week and see. Nobody knows the future anyway.
- globalise83 3y ago
  You should look into decision analysis with decision trees - this is a nice article on how to do it: https://hbr.org/1964/07/decision-trees-for-decision-making https://hbr.org/1964/07/decision-trees-for-decision-making
- barfbagginus 3y ago
  My analysis of the platform risk of the ChatGPT plug-in store depends on 1) risk ideas like Failure Modes and risk hedging. The outcome in the plug-in store is an example of a well known failure mode of platforms that grow very fast. 2) remaining up to date with trends in the llm literature, so we can for instance predict that major vendors would release multimodal llms 3) being aware of trends in tech platform dynamics. Specifically the trend for "platform enshittification" - a term coined by Corey Doctorow - which is the tendency for tech monopolies to exploit their own users and vendors when they can. I'll talk about each of these a little bit more in depth. General Risk Analysis First I recommend reading academic business and cyber risk analysis textbooks. Not just popular business books like "Anti fragile" or "The Black Swan" (which are good), but also academic books that introduced Bayesian networks and Failure Mode and Effect Analysis. You're not going to get away from some math, but thankfully there are techniques that let you draw situations as network diagrams, which I personally find helps with my natural difficulty with mathematics and symbolic processing. You can do pretty decent risk analysis just using "napkin math" - rough graphical and numerical estimates that approximate risks and can sharpen your intuition. I like to recommend cyber risk analysis too, since it makes sense to be able to accurately assess your organizational cyber risks as an entrepreneur. It shows us how beautiful and interdisciplinary risk management can be, and ideas like adversarial modeling, the attack kill chain, attack sequences, and the attack graph are valuable in ordinary business risk management, like coping with insider threats or adversarial competitors. Technical Observation Observing the development of technology is also important, and with AI we have the advantage that most of the development is happening in publicly accessible research papers on arxiv. By following the AI literature and evaluating LLMs and their capabilities, I made these two observations: 1. Multimodal and OCR assisted LLMs were hot in the AI literature early last year, and that approach was very helpful for reducing the hallucination/bullshitting problem - ai's biggest usability problem to date. Some of these papers were from openai themselves. From this we could assume that the major AI vendors would work very hard to integrate this feature, and suppose that third party OCR solutions would get kicked out of the market very quickly once they did. We have seen that with GPT, Gemini, Claude, and more. 2. GPT4 continues to beat the pants off of every single competitor, in the hundred+ benchmark papers I've read, in the public benchmarks like Chatbot Arena, and in my own personal and business use case benchmarking. There's basically no contest on quality, and the result is 100% capture of the market of serious users who depend on GPT4 quality and are sophisticated enough to evaluate AI quality. From this we can assume that OpenAI is well positioned to do everything it can to retain this advantage, even up to compromising developers who think it is safe to use the GPT store to build general improvements for GPT's weaknesses. To gain this technical insight, I highly recommend reading many AI papers, internalizing their insights, and if possible, working hands on with their experimental techniques. Being aware of AI's current weaknesses and benchmarks, and being able to name papers working on overcoming specific problems, allows us to predict what might be coming next from big vendors. Platform Risk Analysis To gain insight into why large corporations tend to hurt their own developer communities and users when they gain the opportunity, I highly recommend Cory Doctorow's 2023 article, "The Enshittification of Tik Tok". It covers a great deal of other examples besides - including Apple and Google's app stores which are relevant to understanding the outcome for the GPT plug in store. And it does a very good analysis of user and vendor exploiting strategies on Tik Tok. Doctorow also writes a freely available book/audiobook, "The Internet Con: Seizing The Means of Comoutation", where he analyzes the issue and its contributing factors, and proposes a solution based on regulations. While his solution - heavy regulation - doesn't seem that realistic, it's a good analysis, especially for entrepreneurs who might want to build on top of platforms that have this form of risk. Finally, here's a paper that builds on this idea and analyzes how platforms are able to use their algorithmic control over the users attention to increasingly disadvantage a variety of stakeholders, including users, vendors, and advertisers: https://www.cambridge.org/core/services/aop-cambridge-core/content/view/D85FE41F6CF99FC57DDFB2B2B63491C5/S2632324924000014a.pdf/algorithmic-attention-rents-a-theory-of-digital-platform-market-power.pdf https://www.cambridge.org/core/services/aop-cambridge-core/c... Conclusion and Personal Strategy I recommend a variety of sources including literature on risk management itself, literature having to do with the technology itself in a large variety of application domains, and being aware of market trends and forces, like the pressure to take advantage of non-financial stakeholders once a platform reaches a market dominance. My general AI business strategy is to focus on my sub niche of technical work where AI has not really penetrated - automating 2d spline processing operations that have been ignored by AI researchers due to their focus on point clouds, radiance fields, bitmaps, and meshes, which are more important and applicable to a huge variety of domains. I focus on providing geometric processing services directly to manufacturers dealing with 2d vector data from CMM systems. The CMM vendor I'm specializing in has been lax in adopting automation technology, and the customers I'm selling to are very reluctant to invest in improved CAD software, even when it could reduce their own labor by a factor of three or more. So between those two gaps I have a niche which I have exploited for 3 years now, and which I expect to exist for at least another 3 years. I plan to turn my CAD automation tools into an attractive product by that time, and also hope to advance the state of the art AI techniques in reverse engineering 2D splines from CMM data. I hope that lets me hold on to the niche until 2030. But I also would not be very surprised if my niche gets disrupted by one of the major CAD or CMM vendors in that time. My worst case outcome seems to be that I would have to join a more solidly established corporation as an research manager, strategic analyst, or software engineer. That could be an improvement in my quality of life, given that my current yearly revenues are still under 60k. However I'm going to fight that with all I've got, because I love owning my own company so much! I am an autistic entrepreneur, and owning my own company has been a huge breakthrough in my Independence and ability to avoid burnout. I can get by on as few as 10 hours of labor a week, which would be very hard to find in a corporation. Anyways, I hope that this message has been beneficial to you even though it grew a bit long. I find Risk Management very fun and rewarding to study and apply, and hope you will as well! I wish you luck in all your endeavors!
timetraveller26 3y ago
Is OpenAI going to use the Amazon playbook of compete with the more popular products? Is not publishing anything on the store in the first place the best way for risk avoidance?
- dragonwriter 3y ago
  > Is OpenAI going to use the Amazon playbook of compete with the more popular products? They overtly target universal capability for the model itself, it’d be surprising if they didn’t, within that, prioritize functionality that has demonstrated that it is in demand for use with the other existing model functionalities, which successful GPTs or plugins clearly demonstrates.

Areibman 3y ago

We knew the time would come, but we built ChatOCR in a week. If we overthought the time horizon problem, we'd have 0 users and $0. But also, GPT-4-Vision is multi-modal but does not specifically use OCR. Our tool is used mostly to extract text from documents and load it into context, and we still saw growth after OpenAI built this feature into ChatGPT.

nextworddev 3y ago
$20k for a week’s work seems more than good, no?
- sangnoir 3y ago
  Whose week's work? One person, a small team (<5), or a large organization? At some point on that axis, the RoI becomes negative.
  - guiambros 3y ago
    You can't get a large organization to move fast and ship something in a week. So this was either a solo developer en devour, or at max low single digits.
barfbagginus 3y ago
Thank you for sharing your risk thinking here. I like that you are risk hungry, not prone to hesitation, and you achieved revenue on a shoe string budget. I'm curious about the idea that your OCR-generated context might give better recall than whatever GPT uses to parse PDFs into context. That would support the idea that it is discoverability, and not feature parity, that is killing the product. Can we discuss benchmarks that show conclusively that your solution does better on QA accuracy with respect to a PDF, as compared to GPT4's own PDF mode? Also I'm curious about the trajectory of active users after the feature dropped from OAI. Was there an inflection point where new active users per month went negative? How long did it take before that inflection happened? Do you still have enough users to keep the service up and gathering revenue, or do you plan to mothball? Do you have any plans to pivot the offering to other markets?

yaj54 3y ago

What resources do you suggest for learning how to do this kind of risk analysis well?

vasco 3y ago
A crystal ball. Most of the times you should do what they did, build a first version in a week and see. Nobody knows the future anyway.
globalise83 3y ago
You should look into decision analysis with decision trees - this is a nice article on how to do it: https://hbr.org/1964/07/decision-trees-for-decision-making https://hbr.org/1964/07/decision-trees-for-decision-making
barfbagginus 3y ago
My analysis of the platform risk of the ChatGPT plug-in store depends on 1) risk ideas like Failure Modes and risk hedging. The outcome in the plug-in store is an example of a well known failure mode of platforms that grow very fast. 2) remaining up to date with trends in the llm literature, so we can for instance predict that major vendors would release multimodal llms 3) being aware of trends in tech platform dynamics. Specifically the trend for "platform enshittification" - a term coined by Corey Doctorow - which is the tendency for tech monopolies to exploit their own users and vendors when they can. I'll talk about each of these a little bit more in depth. General Risk Analysis First I recommend reading academic business and cyber risk analysis textbooks. Not just popular business books like "Anti fragile" or "The Black Swan" (which are good), but also academic books that introduced Bayesian networks and Failure Mode and Effect Analysis. You're not going to get away from some math, but thankfully there are techniques that let you draw situations as network diagrams, which I personally find helps with my natural difficulty with mathematics and symbolic processing. You can do pretty decent risk analysis just using "napkin math" - rough graphical and numerical estimates that approximate risks and can sharpen your intuition. I like to recommend cyber risk analysis too, since it makes sense to be able to accurately assess your organizational cyber risks as an entrepreneur. It shows us how beautiful and interdisciplinary risk management can be, and ideas like adversarial modeling, the attack kill chain, attack sequences, and the attack graph are valuable in ordinary business risk management, like coping with insider threats or adversarial competitors. Technical Observation Observing the development of technology is also important, and with AI we have the advantage that most of the development is happening in publicly accessible research papers on arxiv. By following the AI literature and evaluating LLMs and their capabilities, I made these two observations: 1. Multimodal and OCR assisted LLMs were hot in the AI literature early last year, and that approach was very helpful for reducing the hallucination/bullshitting problem - ai's biggest usability problem to date. Some of these papers were from openai themselves. From this we could assume that the major AI vendors would work very hard to integrate this feature, and suppose that third party OCR solutions would get kicked out of the market very quickly once they did. We have seen that with GPT, Gemini, Claude, and more. 2. GPT4 continues to beat the pants off of every single competitor, in the hundred+ benchmark papers I've read, in the public benchmarks like Chatbot Arena, and in my own personal and business use case benchmarking. There's basically no contest on quality, and the result is 100% capture of the market of serious users who depend on GPT4 quality and are sophisticated enough to evaluate AI quality. From this we can assume that OpenAI is well positioned to do everything it can to retain this advantage, even up to compromising developers who think it is safe to use the GPT store to build general improvements for GPT's weaknesses. To gain this technical insight, I highly recommend reading many AI papers, internalizing their insights, and if possible, working hands on with their experimental techniques. Being aware of AI's current weaknesses and benchmarks, and being able to name papers working on overcoming specific problems, allows us to predict what might be coming next from big vendors. Platform Risk Analysis To gain insight into why large corporations tend to hurt their own developer communities and users when they gain the opportunity, I highly recommend Cory Doctorow's 2023 article, "The Enshittification of Tik Tok". It covers a great deal of other examples besides - including Apple and Google's app stores which are relevant to understanding the outcome for the GPT plug in store. And it does a very good analysis of user and vendor exploiting strategies on Tik Tok. Doctorow also writes a freely available book/audiobook, "The Internet Con: Seizing The Means of Comoutation", where he analyzes the issue and its contributing factors, and proposes a solution based on regulations. While his solution - heavy regulation - doesn't seem that realistic, it's a good analysis, especially for entrepreneurs who might want to build on top of platforms that have this form of risk. Finally, here's a paper that builds on this idea and analyzes how platforms are able to use their algorithmic control over the users attention to increasingly disadvantage a variety of stakeholders, including users, vendors, and advertisers: https://www.cambridge.org/core/services/aop-cambridge-core/content/view/D85FE41F6CF99FC57DDFB2B2B63491C5/S2632324924000014a.pdf/algorithmic-attention-rents-a-theory-of-digital-platform-market-power.pdf https://www.cambridge.org/core/services/aop-cambridge-core/c... Conclusion and Personal Strategy I recommend a variety of sources including literature on risk management itself, literature having to do with the technology itself in a large variety of application domains, and being aware of market trends and forces, like the pressure to take advantage of non-financial stakeholders once a platform reaches a market dominance. My general AI business strategy is to focus on my sub niche of technical work where AI has not really penetrated - automating 2d spline processing operations that have been ignored by AI researchers due to their focus on point clouds, radiance fields, bitmaps, and meshes, which are more important and applicable to a huge variety of domains. I focus on providing geometric processing services directly to manufacturers dealing with 2d vector data from CMM systems. The CMM vendor I'm specializing in has been lax in adopting automation technology, and the customers I'm selling to are very reluctant to invest in improved CAD software, even when it could reduce their own labor by a factor of three or more. So between those two gaps I have a niche which I have exploited for 3 years now, and which I expect to exist for at least another 3 years. I plan to turn my CAD automation tools into an attractive product by that time, and also hope to advance the state of the art AI techniques in reverse engineering 2D splines from CMM data. I hope that lets me hold on to the niche until 2030. But I also would not be very surprised if my niche gets disrupted by one of the major CAD or CMM vendors in that time. My worst case outcome seems to be that I would have to join a more solidly established corporation as an research manager, strategic analyst, or software engineer. That could be an improvement in my quality of life, given that my current yearly revenues are still under 60k. However I'm going to fight that with all I've got, because I love owning my own company so much! I am an autistic entrepreneur, and owning my own company has been a huge breakthrough in my Independence and ability to avoid burnout. I can get by on as few as 10 hours of labor a week, which would be very hard to find in a corporation. Anyways, I hope that this message has been beneficial to you even though it grew a bit long. I find Risk Management very fun and rewarding to study and apply, and hope you will as well! I wish you luck in all your endeavors!

timetraveller26 3y ago

Is OpenAI going to use the Amazon playbook of compete with the more popular products? Is not publishing anything on the store in the first place the best way for risk avoidance?

dragonwriter 3y ago
> Is OpenAI going to use the Amazon playbook of compete with the more popular products? They overtly target universal capability for the model itself, it’d be surprising if they didn’t, within that, prioritize functionality that has demonstrated that it is in demand for use with the other existing model functionalities, which successful GPTs or plugins clearly demonstrates.