6 ms·

I used to work at Wolfram Research on Wolfram Alpha's backend, and one of the most challenging technical problems we faced was free-form input. Although W|A has

I used to work at Wolfram Research on Wolfram Alpha's backend, and one of the most challenging technical problems we faced was free-form input. Although W|A has by no means perfected this, this is hardly an alternative - queries like "factor the number 100" fail because there is only the beginnings of a free-form transformer. Obviously, the usefulness of being able to answer "factor one hundred" is questionable, but W|A solved it out of Stephen Wolfram's aspiration to be able to "compute everything". Right now, this is just a programming language you can run on the web. The other thing is that W|A has trillions of data points and what I am only allowed to describe as the beginnings of a semantic network for inferring relations between them. It was a vastly overcomplicated system that was difficult to work with, so I am quite confident that some day there will be an open-source alternative that anyone can contribute to. The other thing to take into account is how this affects the future. Wolfram Research thought they were going to "disrupt the calculator" (I heard this ridiculous statement once at a meeting). In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery (I know this because there was a big TV in the break room that would keep displaying things that people searched on Wolfram Alpha). Is it really that useful to be able to have a computer give you an answer to "I have two apples, Jill has three apples. How many apples do we both have?" http://www.wolframalpha.com/input/?i=I+have+two+apples%2C+Jill+has+three+apples.+How+many+apples+do+we+both+have%3F http://www.wolframalpha.com/input/?i=I+have+two+apples%2C+Ji... Or is it more useful to make something that can take in symptoms of your current ailment and tell you which disease you are most likely to have? Wolfram Alpha does this as well. http://www.wolframalpha.com/input/?i=I+have+a+fever+and+a+runny+nose http://www.wolframalpha.com/input/?i=I+have+a+fever+and+a+ru... Although the results are difficult to interpret. In my time at Wolfram Research, I was certainly convinced by the idea of knowledge engines and their ultimate emergence, but I think the way this will be accomplished is in more in a Google-esque fashion where their knowledge engine results are displayed alongside a real search algorithm. Best of luck to the people on this project, I hope you make the first step into creating an open source knowledge engine.

mh_yam 13y ago
"In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery" In advanced math classes (upper undergraduate or graduate) it is almost impossible to check results or do a complex operation on a simple calculator. In many cases I have to turn to a tool like Mathematica/Wolfram Alpha. I have a W|A Pro account and it has worked wonders for me. For example, entering "integral from 0 to infinity of (ye^(-y)((-1/y)(e^(-t))+1/y)) with respect to y" into Wolfram Alpha is so much easier than doing the same with a TI-89. I can copy/paste and adjust very easily and the software produces multiple interpretations/representations which is very useful.
- runarberg 13y ago
  I use IPython/SymPy for the same purpose. It is nice to be able to do these calculations on my local machine, in stead of relying on an internet conection. Plus I get the powers of Python to back me up if I want to do anything more compilcated.
  - shawn-furyan 13y ago
    It's not nothing, however, that Wolfram Alpha works spectacularly well on a phone. The mobile app even has a very useful keyboard that has common math symbols. With mobile data, at any given time, I'm much more likely to have a computing device with an internet connection than I am to have one with a useable programming keyboard. Not that I'm against the progression of these tools, far from it, I just think that wolfram alpha does a better job at math homework type problems for most people most of the time. If you have more complicated modeling/statistical/etc. work, then bring out the bigger guns.
    - mh_yam 13y ago
      I used the Wolfram Alpha iPad app in real-time during mathematics / statistics lectures to check the notes on the blackboard and to try out alternatives. Very useful in the classroom.
      - runarberg 13y ago
        I see that some professors (not in my school though) publish they’re lecture notes as IPython notebooks. That is super clever and allows interactivity with the lecture material. I don’t know the mobile/tablet support of IPython notebook however.
- primitivesuave 13y ago
  Your comment reminded me of a very interesting observation I had when I was looking at Wolfram Alpha analytics - for some reason, the number of queries drops dramatically around mid-December, and only returns to the original volume around mid-January. I was puzzled by how this phenomenon repeated itself every year. It took me a minute to realize that during that time, every college student is on vacation.
JumpCrisscross 13y ago
>In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery I would expect the significance of queries on any free engine to adhere to a steep Pareto distribution. For what it's worth I, as an algebraic thinker, find Wolfram Alpha's symbolic manipulation and easy syntax refreshingly useful.
_delirium 13y ago
Out of curiosity, I ran your three examples on Google and Bing, and got: 1. factor the number 100: Bing intercepts it and gives you 2^2 x 5^2, while Google doesn't 2. I have two apples etc.: neither one handles this specially 3. I have a fever and a runny nose: Google intercepts it and suggests several ailments (see https://support.google.com/websearch/answer/2364942?p=g_symp https://support.google.com/websearch/answer/2364942?p=g_symp), while Bing doesn't
- primitivesuave 13y ago
  The symptom-to-ailment mapping is actually a quite complex problem that neither Google nor W|A have thoroughly solved, but Google provides results in a much easier-to-read format.
Jonovono 13y ago
What language does WA use? Is the natural language interpretation run completely by Mathematica? Also, would you recommend someone interested in making a natural language project look into using Mathematica for that?
- primitivesuave 13y ago
  A large part of the natural language processing is indeed done by Mathematica, and last time I looked I believe there were about 15 million lines of Mathematica code in the main repository. Note that this massive number is largely the result of the multitude of Mathematica scripts used to insert raw data and relations into the database. Just based on glancing at folder sizes, I'd estimate that around 40% of the repository is code for parsing, so around 6 million lines of Mathematica code. Note that lines of Mathematica code tend to do a lot of processing, so this would be the equivalent of many times more lines in another language. It is quite an interesting process hooking in a new feature to Wolfram Alpha, and some developers described it as the "mud bowl" because when you break things, you just had to throw more mud at it. I'm not allowed to disclose details about the technology stack, but I can say that the database querying functionality was kept separate from the actual parsing and semantic analysis, and was implemented in a different language. If you're interested in NLP, which by the way is a wonderful and exciting field with mysteries abound, Mathematica is indeed a great way to get started quickly. Although I recommend to everyone to do their work in an open-source-able way with a popular language like Python or Java, I built a Swahili translator during my freshman year of college with Mathematica. Here it is on github: https://github.com/keshavsaharia/Swahili-Translator https://github.com/keshavsaharia/Swahili-Translator Best of luck, feel free to shoot me an email at the email address listed on my github if you have any questions or need help getting started.
  - siddboots 13y ago
    > I'm not allowed to disclose details about the technology stack, but I can say that the database querying functionality was kept separate from the actual parsing and semantic analysis, and was implemented in a different language. I'm definitely curious about this. Are you allowed to disclose which language?
keithpeter 13y ago
"...one of the most challenging technical problems we faced was free-form input." The attempt is greatly appreciated. I can fully see that this is a hard problem.
ch 13y ago
I must admit I was somewhat disappointed that the first query did not use icons of apples in the Manipulatives illustration section!
cowpig 13y ago
" In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery" For me, this is because WA starts asking me for money when I try to use it for useful things. I suspect an open-source version will have different numbers.
mathattack 13y ago
Thank you for sharing. It's interesting (and perhaps obvious in hindsight) that Wolfram has great math in the background, but struggles with the freeform interface.
nl 13y ago
the first step into creating an open source knowledge engine I'm working on this too, but I'm tackling it from the non-math side (ie, NLP+Knowledge Graph). one of the most challenging technical problems we faced was free-form input Yes, it's a horrible problem(!) I'm using Quepy[1] (which in turn uses NLTK), and it does a decent job. It's still not automatically general purpose (you need to write code to map classes of queries), but it can handle questions like "Who directed The Social Network"[2]. The other thing is that W|A has trillions of data points and what I am only allowed to describe as the beginnings of a semantic network for inferring relations between them. It was a vastly overcomplicated system that was difficult to work with, so I am quite confident that some day there will be an open-source alternative that anyone can contribute to. This is interesting to me (for obvious reasons). Can you expand on what made it so complicated (I assume beyond the standard RDF-style inference engines)? [1] http://quepy.machinalis.com/ http://quepy.machinalis.com/ [2] http://quepy.machinalis.com/#Who%20directed%20The%20Social%20Network http://quepy.machinalis.com/#Who%20directed%20The%20Social%2...?
- mindcrime 13y ago
  I'm working on this too, Have you looked at any of the research to come out of http://start.csail.mit.edu/index.php http://start.csail.mit.edu/index.php? I've also been working with Quepy a bit lately. Very cool stuff. Are you able to comment at all on what you're working on, or is it "super secret stealth mode" stuff? For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not. If you're not in super-secret stealth mode and ever want to compare notes to talk about this stuff offline, feel free to shoot me an email.
  - nl 13y ago
    Not especially super-stealth, but it's not really ready for public consumption. Basically I've been glueing lots of pieces pre-existing software together, sticking webservice front ends on them and making them work together. It's all Dockerfied so things can be run separately. For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not. Very similar here. I'm (currently) using DBPedia dumps, loaded into Jena. I'm experimenting with content extraction (for eg CIA Factbook). I have the Quepy->SPARQL mapping working, though. I'll send you an email.
topynate 13y ago
"I have two mangoes, Jill has three mangoes. How many mangoes do we both have?" http://www.wolframalpha.com/input/?i=I+have+two+mangoes%2C+Jill+has+three+mangoes.+How+many+mangoes+do+we+both+have%3F http://www.wolframalpha.com/input/?i=I+have+two+mangoes%2C+J... Doesn't work, even though Wolfram Alpha knows what mangoes are enough to give me the nutritional value of three of them as its response instead. 'Oranges' as the noun works quickly. 'Pens' doesn't work at all, so it's hardly surprising that 'doowats' also fails. Surprisingly, even 'pears' is a fruit too far. On the other hand, "I have two apples and one orange, Jill has three oranges. How many oranges do we both have?" works very well. So I have to ask, is this just a trick? I mean, did you program it to handle apples and oranges specifically, without attempting to do any sort of semantic comprehension? Because this is a big failing of Wolfram Alpha for me. It 'knows' things but it doesn't know them. It knows that a mango has 84 calories but not that it's countable. Or perhaps it does know that and that knowledge just doesn't propagate to more complex queries.
- e12e 13y ago
  Yeah, it doesn't appear to be very advanced. I tried: I have two mangoes, Jill has three mangoes. Alan has three oranges, and I have two apples and a carrot. How many mangoes do we have? How many fruits do we all have? How long can we survive? One would think it did some basic parsing, and managed to file away three oranges and two apples in a parse tree of some kind -- and if those were associated with "fruits" -- it should be able to answer? And if it were able to answer, one might be able to have it hand out advice on diets ("Give me 30 examples of a 3000 calorie diet featuring no red meat") and a lot of other things that would be "easy" to answer based on ingesting some pretty standard databases.
  - primitivesuave 13y ago
    I see a couple issues with handling your "30 examples of 3000 calorie diet with no red meat" query. One is the complexity of finding a subset of all the food items in the Wolfram Alpha database that add up to around 3000, you could certainly make an algorithm to do this but all queries time out after a few seconds so it is likely to fail. The other is that it would have to understand which foods constitute red meat, which it doesn't (try searching "red meat food", the only red meat it knows is a movie by that name). Even if it did, you would need to categorize all foods so the query worked with "3000 calorie diet with no vegetables" and so on. Obviously if it can solve word problems about apples but can't solve the same problems about mangoes, computation like this is far beyond what they will accomplish in many years.
No1 13y ago
> but I think the way this will be accomplished is in more in a Google-esque fashion where their knowledge engine results are displayed alongside a real search algorithm. http://www.goofram.com/ http://www.goofram.com/ does exactly that, albeit somewhat crudely. You can get some unexpectedly interesting stuff on the Wolfram Alpha side of things, though usually it's irrelevant unless you're "thinking" in Wolfram query mode.
jules 13y ago
When I use Wolfram Alpha, I am always terribly frustrated by the natural language input. This "natural language" only works when the exact query you put in was anticipated by an engineer at Wolfram. I'd rather have auto complete where I can type "least common multiple" and it shows me that the function is lcm(n,k).

mh_yam 13y ago

"In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery" In advanced math classes (upper undergraduate or graduate) it is almost impossible to check results or do a complex operation on a simple calculator. In many cases I have to turn to a tool like Mathematica/Wolfram Alpha. I have a W|A Pro account and it has worked wonders for me. For example, entering "integral from 0 to infinity of (ye^(-y)((-1/y)(e^(-t))+1/y)) with respect to y" into Wolfram Alpha is so much easier than doing the same with a TI-89. I can copy/paste and adjust very easily and the software produces multiple interpretations/representations which is very useful.

runarberg 13y ago
I use IPython/SymPy for the same purpose. It is nice to be able to do these calculations on my local machine, in stead of relying on an internet conection. Plus I get the powers of Python to back me up if I want to do anything more compilcated.
- shawn-furyan 13y ago
  It's not nothing, however, that Wolfram Alpha works spectacularly well on a phone. The mobile app even has a very useful keyboard that has common math symbols. With mobile data, at any given time, I'm much more likely to have a computing device with an internet connection than I am to have one with a useable programming keyboard. Not that I'm against the progression of these tools, far from it, I just think that wolfram alpha does a better job at math homework type problems for most people most of the time. If you have more complicated modeling/statistical/etc. work, then bring out the bigger guns.
  - mh_yam 13y ago
    I used the Wolfram Alpha iPad app in real-time during mathematics / statistics lectures to check the notes on the blackboard and to try out alternatives. Very useful in the classroom.
    - runarberg 13y ago
      I see that some professors (not in my school though) publish they’re lecture notes as IPython notebooks. That is super clever and allows interactivity with the lecture material. I don’t know the mobile/tablet support of IPython notebook however.
primitivesuave 13y ago
Your comment reminded me of a very interesting observation I had when I was looking at Wolfram Alpha analytics - for some reason, the number of queries drops dramatically around mid-December, and only returns to the original volume around mid-January. I was puzzled by how this phenomenon repeated itself every year. It took me a minute to realize that during that time, every college student is on vacation.

JumpCrisscross 13y ago

>In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery I would expect the significance of queries on any free engine to adhere to a steep Pareto distribution. For what it's worth I, as an algebraic thinker, find Wolfram Alpha's symbolic manipulation and easy syntax refreshingly useful.

_delirium 13y ago

Out of curiosity, I ran your three examples on Google and Bing, and got: 1. factor the number 100: Bing intercepts it and gives you 2^2 x 5^2, while Google doesn't 2. I have two apples etc.: neither one handles this specially 3. I have a fever and a runny nose: Google intercepts it and suggests several ailments (see https://support.google.com/websearch/answer/2364942?p=g_symp https://support.google.com/websearch/answer/2364942?p=g_symp), while Bing doesn't

primitivesuave 13y ago
The symptom-to-ailment mapping is actually a quite complex problem that neither Google nor W|A have thoroughly solved, but Google provides results in a much easier-to-read format.

Jonovono 13y ago

What language does WA use? Is the natural language interpretation run completely by Mathematica? Also, would you recommend someone interested in making a natural language project look into using Mathematica for that?

primitivesuave 13y ago
A large part of the natural language processing is indeed done by Mathematica, and last time I looked I believe there were about 15 million lines of Mathematica code in the main repository. Note that this massive number is largely the result of the multitude of Mathematica scripts used to insert raw data and relations into the database. Just based on glancing at folder sizes, I'd estimate that around 40% of the repository is code for parsing, so around 6 million lines of Mathematica code. Note that lines of Mathematica code tend to do a lot of processing, so this would be the equivalent of many times more lines in another language. It is quite an interesting process hooking in a new feature to Wolfram Alpha, and some developers described it as the "mud bowl" because when you break things, you just had to throw more mud at it. I'm not allowed to disclose details about the technology stack, but I can say that the database querying functionality was kept separate from the actual parsing and semantic analysis, and was implemented in a different language. If you're interested in NLP, which by the way is a wonderful and exciting field with mysteries abound, Mathematica is indeed a great way to get started quickly. Although I recommend to everyone to do their work in an open-source-able way with a popular language like Python or Java, I built a Swahili translator during my freshman year of college with Mathematica. Here it is on github: https://github.com/keshavsaharia/Swahili-Translator https://github.com/keshavsaharia/Swahili-Translator Best of luck, feel free to shoot me an email at the email address listed on my github if you have any questions or need help getting started.
- siddboots 13y ago
  > I'm not allowed to disclose details about the technology stack, but I can say that the database querying functionality was kept separate from the actual parsing and semantic analysis, and was implemented in a different language. I'm definitely curious about this. Are you allowed to disclose which language?

keithpeter 13y ago

"...one of the most challenging technical problems we faced was free-form input." The attempt is greatly appreciated. I can fully see that this is a hard problem.

ch 13y ago

I must admit I was somewhat disappointed that the first query did not use icons of apples in the Manipulatives illustration section!

cowpig 13y ago

" In reality, Wolfram Alpha queries are more often for the sake of fun than for the sake of discovery" For me, this is because WA starts asking me for money when I try to use it for useful things. I suspect an open-source version will have different numbers.

mathattack 13y ago

Thank you for sharing. It's interesting (and perhaps obvious in hindsight) that Wolfram has great math in the background, but struggles with the freeform interface.

nl 13y ago

the first step into creating an open source knowledge engine I'm working on this too, but I'm tackling it from the non-math side (ie, NLP+Knowledge Graph). one of the most challenging technical problems we faced was free-form input Yes, it's a horrible problem(!) I'm using Quepy[1] (which in turn uses NLTK), and it does a decent job. It's still not automatically general purpose (you need to write code to map classes of queries), but it can handle questions like "Who directed The Social Network"[2]. The other thing is that W|A has trillions of data points and what I am only allowed to describe as the beginnings of a semantic network for inferring relations between them. It was a vastly overcomplicated system that was difficult to work with, so I am quite confident that some day there will be an open-source alternative that anyone can contribute to. This is interesting to me (for obvious reasons). Can you expand on what made it so complicated (I assume beyond the standard RDF-style inference engines)? [1] http://quepy.machinalis.com/ http://quepy.machinalis.com/ [2] http://quepy.machinalis.com/#Who%20directed%20The%20Social%20Network http://quepy.machinalis.com/#Who%20directed%20The%20Social%2...?

mindcrime 13y ago
I'm working on this too, Have you looked at any of the research to come out of http://start.csail.mit.edu/index.php http://start.csail.mit.edu/index.php? I've also been working with Quepy a bit lately. Very cool stuff. Are you able to comment at all on what you're working on, or is it "super secret stealth mode" stuff? For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not. If you're not in super-secret stealth mode and ever want to compare notes to talk about this stuff offline, feel free to shoot me an email.
- nl 13y ago
  Not especially super-stealth, but it's not really ready for public consumption. Basically I've been glueing lots of pieces pre-existing software together, sticking webservice front ends on them and making them work together. It's all Dockerfied so things can be run separately. For us, we already do semantic concept extraction using Apache Stanbol, against content that flows into our enterprise social network product, and then store the associated triples in an RDF triplestore. We have a primitive search feature exposed, which lets you query using SPARQL, but realistically, we know "normals" will never, ever, ever, ever write SPARQL queries, so the big push is to do automated translation from natural language (even if it's a slightly restricted natural language) into SPARQL so users don't have to think about triples and what-not. Very similar here. I'm (currently) using DBPedia dumps, loaded into Jena. I'm experimenting with content extraction (for eg CIA Factbook). I have the Quepy->SPARQL mapping working, though. I'll send you an email.

topynate 13y ago

"I have two mangoes, Jill has three mangoes. How many mangoes do we both have?" http://www.wolframalpha.com/input/?i=I+have+two+mangoes%2C+Jill+has+three+mangoes.+How+many+mangoes+do+we+both+have%3F http://www.wolframalpha.com/input/?i=I+have+two+mangoes%2C+J... Doesn't work, even though Wolfram Alpha knows what mangoes are enough to give me the nutritional value of three of them as its response instead. 'Oranges' as the noun works quickly. 'Pens' doesn't work at all, so it's hardly surprising that 'doowats' also fails. Surprisingly, even 'pears' is a fruit too far. On the other hand, "I have two apples and one orange, Jill has three oranges. How many oranges do we both have?" works very well. So I have to ask, is this just a trick? I mean, did you program it to handle apples and oranges specifically, without attempting to do any sort of semantic comprehension? Because this is a big failing of Wolfram Alpha for me. It 'knows' things but it doesn't know them. It knows that a mango has 84 calories but not that it's countable. Or perhaps it does know that and that knowledge just doesn't propagate to more complex queries.

e12e 13y ago
Yeah, it doesn't appear to be very advanced. I tried: I have two mangoes, Jill has three mangoes. Alan has three oranges, and I have two apples and a carrot. How many mangoes do we have? How many fruits do we all have? How long can we survive? One would think it did some basic parsing, and managed to file away three oranges and two apples in a parse tree of some kind -- and if those were associated with "fruits" -- it should be able to answer? And if it were able to answer, one might be able to have it hand out advice on diets ("Give me 30 examples of a 3000 calorie diet featuring no red meat") and a lot of other things that would be "easy" to answer based on ingesting some pretty standard databases.
- primitivesuave 13y ago
  I see a couple issues with handling your "30 examples of 3000 calorie diet with no red meat" query. One is the complexity of finding a subset of all the food items in the Wolfram Alpha database that add up to around 3000, you could certainly make an algorithm to do this but all queries time out after a few seconds so it is likely to fail. The other is that it would have to understand which foods constitute red meat, which it doesn't (try searching "red meat food", the only red meat it knows is a movie by that name). Even if it did, you would need to categorize all foods so the query worked with "3000 calorie diet with no vegetables" and so on. Obviously if it can solve word problems about apples but can't solve the same problems about mangoes, computation like this is far beyond what they will accomplish in many years.

No1 13y ago

> but I think the way this will be accomplished is in more in a Google-esque fashion where their knowledge engine results are displayed alongside a real search algorithm. http://www.goofram.com/ http://www.goofram.com/ does exactly that, albeit somewhat crudely. You can get some unexpectedly interesting stuff on the Wolfram Alpha side of things, though usually it's irrelevant unless you're "thinking" in Wolfram query mode.

jules 13y ago

When I use Wolfram Alpha, I am always terribly frustrated by the natural language input. This "natural language" only works when the exact query you put in was anticipated by an engineer at Wolfram. I'd rather have auto complete where I can type "least common multiple" and it shows me that the function is lcm(n,k).