8 ms·

Speech and Language Processing, 3rd ed. draft (2017)

219 points by panyang 9y ago

contingencies 9y ago
Often this stuff is used for surveillance. If you choose to study this area, please be careful how you apply your knowledge. There are plenty of positive ways to use it: contributing content classification systems to sci-hub or libgen, building tools for the disabled, automating multilingual visual design aesthetics with computational linguistics and machine learning...
- ngcc_hk 9y ago
  Whilst I understand the concern but just Internet is now for control and tracking (chi-na is more obvious but post someone we all know better), we cannot stop the wheel. Not knowing is even worst.
  - amelius 9y ago
    We need an equivalent of the hippocratic oath for software engineers: I swear by Hephaestus, the ugly god of crafts, that I shall not use my skills for controlling and tracking ...
    - myaso 9y ago
      Great, these who want to build these things will simply hire people who don't subscribe to this oath ... And now the organization will be populated by a whole tribe of people who have no qualms about doing this -- does that sound like something that will make things better? Aside: the book looked interesting. Skimmed it, beautiful typesetting, not gratuitously mathematical, looked readable -- I may be wrong, but it's just a exercise of adding awareness of it's existence to me.
      - amelius 9y ago
        > Great, these who want to build these things will simply hire people who don't subscribe to this oath ... How many doctors are willing to give up their title to do unethical stuff? An ethical code will change people's perception, and that's a big thing. It will morph the perception of Google and Facebook from cool tech companies into dark places where no decent people want to work; unless of course these companies are willing to change. And there will be a huge effect on politics, other businesses, and media.
        myaso 9y ago
        Um, how are they supposed to make money if they don't track you to a certain extent? Do they have flaws? Yes -- it's not funny what kind of things you can target for based AdWords, their internal staff probably aren't creative enough to think up ways in which their tools can be used to inflict misery. Software engineers aren't soldiers, doctors, or civil servants what kind of ethical code do you expect? How will people be held accountable to it? What will happen to these who make transgressions?
        randcraw 9y ago
        And how are cigarette companies going to make money if they are hindered in addicting you and giving you cancer? Legally of course. Software engineers are surrogates for ALL jobs because software is used everywhere. When we serve medicine, we should be obliged to take the Hippocratic oath. When we serve the military or intel agencies (with a security clearance), we ARE obliged to take an formal oath of secrecy. IMHO, many other S/W roles bear no less responsibility. I think it's time that we software infrastructuralists take our role behind the scenes as seriously as those on the front lines do.
        myaso 9y ago
        Really? As a former smoker I don't place cigarettes and software in even remotely the same category. Ok -- question. How will this be enforced? Legal/Medical/Eng have licenses to practice you can lose these licenses. I don't need anybody's approval to write code, all the tools are available for free. You are naive if you think some feel good words on paper can make this work without some force backing these words.
      - matt4077 9y ago
        > Great, these who want to build these things will simply hire people who don't subscribe to this oath .. Morally, this argument quite obviously fails because it could be used to justify anything. Practically, we see many professions where people of character have successfully resisted temptation: every government employee who is not corrupt comes to mind. And even if they'll just hire the next guy/gal: If you say no to an offer, you were, by definition, their top choice. That next one down the ladder will therefore be slightly worse, making such businesses less profitable.
        myaso 9y ago
        Top choice based on a search with finite resources, maybe there is a better top choice if they spend a bit more time -- one that is also more agreeable. I'll return with a question -- Is resisting temptation the path of least resistance? What are the odds that the personality trait you describe also overlaps with the person being in a position of enough authority to actually matter?
    - Jedi72 9y ago
      Early in my career (it began after a discussion in a tutorial during my undergrad) I made a choice to never to use my engineering skills to build weapons. I don't believe its a morally justifiable choice in this century. It's not like I have been forced to starve as a result but it did reduce a few opportunities.
      - dominotw 9y ago
        > I don't believe its a morally justifiable choice in this century. Could we say that nuclear(and other) weapons are responsible for the bringing unprecedented world peace to the world?
        glup 9y ago
        Yes, until they don't.
        matt4077 9y ago
        Probably not. There are many countries not engaging in warfare even though they do not have nuclear weapons. Belgium and the Netherlands, for example, do not possess them, nor is there an obvious mechanism why the US' (or Britain's/France's) nuclear arsenal would discourage a war between these nations. The US, meanwhile, has been engaged in wars almost non-stop since WW2. It's "unprecedented world peace" for for only a select few nations. And, most importantly, the costs of failure have gone up dramatically with the invention of nuclear weapons. The Cuban Missile Crises is just one of several near-catastrophic events. Sooner or later, we will get a madman in a position of power, fall victim to a misunderstanding, or suffer a technical failure. And that'll be that...
        dominotw 9y ago
        https://persuademepolitics.com/2016/03/23/now-the-most-peaceful-time-in-world-history/ https://persuademepolitics.com/2016/03/23/now-the-most-peace...
        gspetr 9y ago
        There would be nothing to be gained in that war for either country. You don't need territory (at least at a price that steep) and you certainly don't need slave labor in 2018.
    - pmcjones 9y ago
      See the ACM's code of ethics, which they are in the process of updating: https://dl.acm.org/ft_gateway.cfm?id=3173016&ftid=1936873&dwn=1 https://dl.acm.org/ft_gateway.cfm?id=3173016&ftid=1936873&dw... .
- wdiamond 9y ago
  Fast programming equals surveilance. Programs must be asserted at some point in op cycle. The point of privacy is not about capturing software device neither parsing software device, it is when to use an external(internet) service.
- brian_spiering 9y ago
  Inventing ships also invented shipwreks
jhanschoo 9y ago
On the slim chance that someone here wants to re-implement the unmodified Kneser-Ney algorithm[0], the presentation of it by the book does not account for unknown tokens in the query not in the vocabulary. I extended the recurrence to its natural closure including unknown tokens here [https://github.com/jhanschoo/HMMTagger/blob/master/readme.pdf https://github.com/jhanschoo/HMMTagger/blob/master/readme.pd...]. A straightforward task, but it might take you an hour or two (probably more) otherwise to obtain it and prove its correctness, seeing as I couldn't find an extension in a Google search nor is it described in the original paper as well. I believe that it would likewise be straightforward to extend this to modified Kneser-Ney as well. [0]: The modification of using multiple discount values due to Chen & Goodman is regarded as a more well-behaved smoothing, and more popular today.
hackernewsacct 9y ago
What are good resources for beginners wanting to learn about natural language processing? Are there any good books, tutorials, courses, etc?
- brian_spiering 9y ago
  https://github.com/keon/awesome-nlp https://github.com/keon/awesome-nlp
- kccqzy 9y ago
  The very book that you are commenting on.
- glup 9y ago
  NLTK tutorials are interesting for the absolute basics in Python, e.g. http://www.nltk.org/book/ch01.html http://www.nltk.org/book/ch01.html. The field has moved into advanced Bayesian and neural methods, but NLTK still has a use in pre-processing and thinking about how to work with text. The first thing to recognize, I think, is to realize how hard many of the language tasks are that people do trivially.
hauscraft 9y ago
Friendly small business loans https://www.cityfinances.lv/kredits-uznemumiem/ https://www.cityfinances.lv/kredits-uznemumiem/
lgessler 9y ago
Note that the 3rd edition isn't entirely finished yet. A full table of planned chapters is here: https://web.stanford.edu/~jurafsky/slp3 https://web.stanford.edu/~jurafsky/slp3
imurray 9y ago
Why link to the pdf? The webpage https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/ links directly to the PDF, and gives context and other download options. It's not so easy to go back from the PDF to the webpage. Mods: I'd change the link, and the title to "...3rd Edition draft". Everyone else: please stop linking to PDFs when there is an obvious html page to link to instead.
- wadkar 9y ago
  Good point, moreover it would be easy to read the simple html than wait for a minute to load the PDF
- sctb 9y ago
  Sure thing, we've updated the link from https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf.
JabavuAdams 9y ago
I need a voice activity detection module (VAD) for my wearable computer. Should I roll my own, or use someone else's (open-source). My immediate need is speaker-dependent (just me), but it would be nice if I could offer up a speaker-independent version eventually.
- woodson 9y ago
  Check out this one: https://github.com/wiseman/py-webrtcvad https://github.com/wiseman/py-webrtcvad If this one does not work for your application, perhaps look into simpler ones like the ones used in mobile telephone codecs or in Speex.
  - JabavuAdams 9y ago
    Thanks! This is exactly what I needed. I was able to get it up and running on Raspberry Pi with just "pip3 install webrtcvad", and the quality is at least good enough to get me started.
zerkten 9y ago
I found speech and language processing to be one of the most interesting courses of my degree. Recently I decided to take a look at speech synthesis again and discovered a book by Paul Taylor on this subject (http://svr-www.eng.cam.ac.uk/~pat40/ http://svr-www.eng.cam.ac.uk/~pat40/ and draft PDF at http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf). It is more engineering focused than other books in this area.
- woodson 9y ago
  Note that the book was released 2009 and the draft is from 2007. While most of the content is still very relevant for understanding the basics and challenges in building TTS systems, recent progress in DNN-based synthesis (including WaveNet, GANs, and end-to-end approaches like Tacotron) is obviously not covered.
akditer 9y ago
very good theoretical information

219 points by panyang 9y ago

Often this stuff is used for surveillance. If you choose to study this area, please be careful how you apply your knowledge. There are plenty of positive ways to use it: contributing content classification systems to sci-hub or libgen, building tools for the disabled, automating multilingual visual design aesthetics with computational linguistics and machine learning...

ngcc_hk 9y ago
Whilst I understand the concern but just Internet is now for control and tracking (chi-na is more obvious but post someone we all know better), we cannot stop the wheel. Not knowing is even worst.
- amelius 9y ago
  We need an equivalent of the hippocratic oath for software engineers: I swear by Hephaestus, the ugly god of crafts, that I shall not use my skills for controlling and tracking ...
  - myaso 9y ago
    Great, these who want to build these things will simply hire people who don't subscribe to this oath ... And now the organization will be populated by a whole tribe of people who have no qualms about doing this -- does that sound like something that will make things better? Aside: the book looked interesting. Skimmed it, beautiful typesetting, not gratuitously mathematical, looked readable -- I may be wrong, but it's just a exercise of adding awareness of it's existence to me.
    - amelius 9y ago
      > Great, these who want to build these things will simply hire people who don't subscribe to this oath ... How many doctors are willing to give up their title to do unethical stuff? An ethical code will change people's perception, and that's a big thing. It will morph the perception of Google and Facebook from cool tech companies into dark places where no decent people want to work; unless of course these companies are willing to change. And there will be a huge effect on politics, other businesses, and media.
      - myaso 9y ago
        Um, how are they supposed to make money if they don't track you to a certain extent? Do they have flaws? Yes -- it's not funny what kind of things you can target for based AdWords, their internal staff probably aren't creative enough to think up ways in which their tools can be used to inflict misery. Software engineers aren't soldiers, doctors, or civil servants what kind of ethical code do you expect? How will people be held accountable to it? What will happen to these who make transgressions?
        randcraw 9y ago
        And how are cigarette companies going to make money if they are hindered in addicting you and giving you cancer? Legally of course. Software engineers are surrogates for ALL jobs because software is used everywhere. When we serve medicine, we should be obliged to take the Hippocratic oath. When we serve the military or intel agencies (with a security clearance), we ARE obliged to take an formal oath of secrecy. IMHO, many other S/W roles bear no less responsibility. I think it's time that we software infrastructuralists take our role behind the scenes as seriously as those on the front lines do.
        myaso 9y ago
        Really? As a former smoker I don't place cigarettes and software in even remotely the same category. Ok -- question. How will this be enforced? Legal/Medical/Eng have licenses to practice you can lose these licenses. I don't need anybody's approval to write code, all the tools are available for free. You are naive if you think some feel good words on paper can make this work without some force backing these words.
    - matt4077 9y ago
      > Great, these who want to build these things will simply hire people who don't subscribe to this oath .. Morally, this argument quite obviously fails because it could be used to justify anything. Practically, we see many professions where people of character have successfully resisted temptation: every government employee who is not corrupt comes to mind. And even if they'll just hire the next guy/gal: If you say no to an offer, you were, by definition, their top choice. That next one down the ladder will therefore be slightly worse, making such businesses less profitable.
      - myaso 9y ago
        Top choice based on a search with finite resources, maybe there is a better top choice if they spend a bit more time -- one that is also more agreeable. I'll return with a question -- Is resisting temptation the path of least resistance? What are the odds that the personality trait you describe also overlaps with the person being in a position of enough authority to actually matter?
  - Jedi72 9y ago
    Early in my career (it began after a discussion in a tutorial during my undergrad) I made a choice to never to use my engineering skills to build weapons. I don't believe its a morally justifiable choice in this century. It's not like I have been forced to starve as a result but it did reduce a few opportunities.
    - dominotw 9y ago
      > I don't believe its a morally justifiable choice in this century. Could we say that nuclear(and other) weapons are responsible for the bringing unprecedented world peace to the world?
      - glup 9y ago
        Yes, until they don't.
      - matt4077 9y ago
        Probably not. There are many countries not engaging in warfare even though they do not have nuclear weapons. Belgium and the Netherlands, for example, do not possess them, nor is there an obvious mechanism why the US' (or Britain's/France's) nuclear arsenal would discourage a war between these nations. The US, meanwhile, has been engaged in wars almost non-stop since WW2. It's "unprecedented world peace" for for only a select few nations. And, most importantly, the costs of failure have gone up dramatically with the invention of nuclear weapons. The Cuban Missile Crises is just one of several near-catastrophic events. Sooner or later, we will get a madman in a position of power, fall victim to a misunderstanding, or suffer a technical failure. And that'll be that...
        dominotw 9y ago
        https://persuademepolitics.com/2016/03/23/now-the-most-peaceful-time-in-world-history/ https://persuademepolitics.com/2016/03/23/now-the-most-peace...
        gspetr 9y ago
        There would be nothing to be gained in that war for either country. You don't need territory (at least at a price that steep) and you certainly don't need slave labor in 2018.
  - pmcjones 9y ago
    See the ACM's code of ethics, which they are in the process of updating: https://dl.acm.org/ft_gateway.cfm?id=3173016&ftid=1936873&dwn=1 https://dl.acm.org/ft_gateway.cfm?id=3173016&ftid=1936873&dw... .
wdiamond 9y ago
Fast programming equals surveilance. Programs must be asserted at some point in op cycle. The point of privacy is not about capturing software device neither parsing software device, it is when to use an external(internet) service.
brian_spiering 9y ago
Inventing ships also invented shipwreks

jhanschoo 9y ago

On the slim chance that someone here wants to re-implement the unmodified Kneser-Ney algorithm[0], the presentation of it by the book does not account for unknown tokens in the query not in the vocabulary. I extended the recurrence to its natural closure including unknown tokens here [https://github.com/jhanschoo/HMMTagger/blob/master/readme.pdf https://github.com/jhanschoo/HMMTagger/blob/master/readme.pd...]. A straightforward task, but it might take you an hour or two (probably more) otherwise to obtain it and prove its correctness, seeing as I couldn't find an extension in a Google search nor is it described in the original paper as well. I believe that it would likewise be straightforward to extend this to modified Kneser-Ney as well. [0]: The modification of using multiple discount values due to Chen & Goodman is regarded as a more well-behaved smoothing, and more popular today.

hackernewsacct 9y ago

What are good resources for beginners wanting to learn about natural language processing? Are there any good books, tutorials, courses, etc?

brian_spiering 9y ago
https://github.com/keon/awesome-nlp https://github.com/keon/awesome-nlp
kccqzy 9y ago
The very book that you are commenting on.
glup 9y ago
NLTK tutorials are interesting for the absolute basics in Python, e.g. http://www.nltk.org/book/ch01.html http://www.nltk.org/book/ch01.html. The field has moved into advanced Bayesian and neural methods, but NLTK still has a use in pre-processing and thinking about how to work with text. The first thing to recognize, I think, is to realize how hard many of the language tasks are that people do trivially.

hauscraft 9y ago

Friendly small business loans https://www.cityfinances.lv/kredits-uznemumiem/ https://www.cityfinances.lv/kredits-uznemumiem/

lgessler 9y ago

Note that the 3rd edition isn't entirely finished yet. A full table of planned chapters is here: https://web.stanford.edu/~jurafsky/slp3 https://web.stanford.edu/~jurafsky/slp3

imurray 9y ago

Why link to the pdf? The webpage https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/ links directly to the PDF, and gives context and other download options. It's not so easy to go back from the PDF to the webpage. Mods: I'd change the link, and the title to "...3rd Edition draft". Everyone else: please stop linking to PDFs when there is an obvious html page to link to instead.

wadkar 9y ago
Good point, moreover it would be easy to read the simple html than wait for a minute to load the PDF
sctb 9y ago
Sure thing, we've updated the link from https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf.

JabavuAdams 9y ago

I need a voice activity detection module (VAD) for my wearable computer. Should I roll my own, or use someone else's (open-source). My immediate need is speaker-dependent (just me), but it would be nice if I could offer up a speaker-independent version eventually.

woodson 9y ago
Check out this one: https://github.com/wiseman/py-webrtcvad https://github.com/wiseman/py-webrtcvad If this one does not work for your application, perhaps look into simpler ones like the ones used in mobile telephone codecs or in Speex.
- JabavuAdams 9y ago
  Thanks! This is exactly what I needed. I was able to get it up and running on Raspberry Pi with just "pip3 install webrtcvad", and the quality is at least good enough to get me started.

zerkten 9y ago

I found speech and language processing to be one of the most interesting courses of my degree. Recently I decided to take a look at speech synthesis again and discovered a book by Paul Taylor on this subject (http://svr-www.eng.cam.ac.uk/~pat40/ http://svr-www.eng.cam.ac.uk/~pat40/ and draft PDF at http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf). It is more engineering focused than other books in this area.

woodson 9y ago
Note that the book was released 2009 and the draft is from 2007. While most of the content is still very relevant for understanding the basics and challenges in building TTS systems, recent progress in DNN-based synthesis (including WaveNet, GANs, and end-to-end approaches like Tacotron) is obviously not covered.

akditer 9y ago

very good theoretical information