7 ms·

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about ho

This isn't really JS, it's a purpose built evaluator that's only for evaluating a particular script on YouTube, assuming a huge list of things are true about how YouTube JS is written. Ex. Its got a hard coded list of methods for String, and it doesn't respect prototypes. It only supports creating Date instances, and won't work if you override the global Date. It parses with regexes and implements all operators with python's operator module (which is the wrong type semantics) etc. Nearly none of the semantics of JS are implemented. It's sort of the sandwich categorization problem: If I write a C# "interpreter" in perl thats only 200 lines and just handles string.Join, string.Concat and Console.WriteLine, and it doesn't actually try to implement C# syntax or semantics at all and just uses perl semantics for those operations is it actually C#? :P I say "not a sandwich".

tra3 4y ago
It’s quacks like a duck at midnight, but it’s actually a frog?
Test0129 4y ago
This really isn't fair. Just because it doesn't faithfully implement whatever standard Javascript is on doesn't mean it isn't an interpreter. All an interpreter is is something that executes a script directly rather than requiring compilation. It is a defacto interpreter for a subset of javascript. Nothing more, nothing less. The title could be more clear, however.
- baobabKoodaa 4y ago
  There's a huge difference between an interpreter for "JavaScript" and an interpreter for a "subset of JavaScript".
  - Test0129 4y ago
    Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter. I'll continue to eat downvotes on this because of the pedantry of HN.
    - khazhoux 4y ago
      Technically, it’s only the pedantry of a subset of HN.
      - lupire 4y ago
        It's an interpretation of a subset of the pedantry on HN.
    - jraph 4y ago
      I didn't downvote, but I don't think esprehn is being unfair. Their comment is very informative. They didn't argue that what was implemented is not an interpreter, they did explain why it's not a JavaScript interpreter and not even an interpreter for a subset of JavaScript. It's just a special purpose interpreter suitable for YouTube's code that cannot be re-used for any code that uses the subset that it seems to implement. It's not pedantry (or I'm pedantic). It's a reaction to the title that can lead people to believe that a complete JavaScript interpreter has been written in less than a thousand lines of Python. This reaction is perfectly understandable.
    - chess_buster 4y ago
      I evaluated it with my Pedantic Interpreter which only results in the `pedantic` token.
    - baobabKoodaa 4y ago
      > Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter. It's not a pedantic argument. Based on the title I thought that somebody wrote something akin to V8 in 800 lines of Python. After reading the comments I realized those 800 lines just interpret a particular JavaScript function written by Youtube. Those things are different. Pointing out the fact that they are different is not pedantry. The title is misleading and the comments pointing that out are helpful.
    - blondin 4y ago
      my vote is meaningless and i am sorry about that. but just wanted to let you know that what you said made sense. do not let people get to you. most of us know that a thousand or so lines of code is not a full JavaScript interpreter and cannot be the real thing. there is no argument or conversation to have about it.
- deleted 4y ago
  [deleted]
- blast 4y ago
  esprehn didn't say it isn't an interpreter. They're saying it is an interpreter and what it's interpreting isn't (all of) JS. That's also what you're saying, so you're agreeing with esprehn. Edit: You misunderstood baobabKoodaa in the same way. Nobody is arguing about what constitutes an interpreter, except you. The question is what language is being interpreted. Before accusing someone of pedantry, it would first be good not to completely misread them.
jraph 4y ago
And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.
- sebzim4500 4y ago
  I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?
  - kevingadd 4y ago
    Embedding a whole js engine and then interopping with it from python would be non trivial. Good luck fixing any bugs or corner cases you hit that way. The V8 and spidermonkey embedding apis are both c++ (iirc) and non trivial to use correctly. Having full control like this +simple code is probably lower risk and more maintainable, even if there's the challenge of expanding feature set if scripts change. The alternative would be a console js shell, but those are very different from browsers so that poses it's own challenges.
    - esprehn 4y ago
      Fwiw there are python bindings for QuickJS and Duktape: https://github.com/PetterS/quickjs https://github.com/PetterS/quickjs https://github.com/stefano/pyduktape https://github.com/stefano/pyduktape https://github.com/amol-/dukpy https://github.com/amol-/dukpy I can't speak to the quality of those bindings, but they do seem maintained.
    - em-bee 4y ago
      apparently yt-dlp is somehow calling out to a js engine if available
      - kevingadd 4y ago
        Yeah, it's possible to install v8 or spidermonkey shells and use them to run code - we use them to run parts of the .NET wasm test suite - but they have a bunch of arbitrary limitations, so if you're trying to emulate a browser I'm not sure I'd bet on them. It's certainly going to be easier than a C++ embedding, so it makes sense that they took that route. Another option is to use node, but it also has weird limitations/behaviors when running code.
blast 4y ago
I suppose this means it would be easy for YouTube to fuck with youtube-dl simply by throwing in more features of JS?
- _pn3l 4y ago
  Cat, meet mouse.
  - nyanpasu64 4y ago
    It's unfortunate, https://github.com/mpv-player/mpv/issues/8655#issuecomment-1199733728 https://github.com/mpv-player/mpv/issues/8655#issuecomment-1...: > Youtube now throttles requests of more than 10MB at a time, yt-dlp works around it by making many requests of 10MB using Range HTTP headers (yt-dlp calls it the http-chunk-size), but ffmpeg which does the downloading for mpv doesn't support that yet. I want to change mpv or yt-dlp to support range-based video URLs (eg. appending &range=333999644-335298975&rn=5&rbuf=0 to URLs) which speed up stream seeking and probably eliminate throttling altogether, but I haven't taken the time to look into how to achieve it. For anyone interested, I have an open bug report at https://github.com/mpv-player/mpv/issues/10601 https://github.com/mpv-player/mpv/issues/10601, and have found https://satadalsengupta.github.io/docs/papers/2017_nossdav_youtubedash.pdf https://satadalsengupta.github.io/docs/papers/2017_nossdav_y... describing these parameters.
dang 4y ago
Ok, we've changed this title to shrink the scope of the interpreter. Submitted title was "YouTube-dl has a JavaScript interpreter written in 870 lines of Python".
- ec109685 4y ago
  Hence why HN better than Twitter. The amount of high engagement just plain wrong tweets there are is just sad.
deleted 4y ago
[deleted]

tra3 4y ago

It’s quacks like a duck at midnight, but it’s actually a frog?

Test0129 4y ago

This really isn't fair. Just because it doesn't faithfully implement whatever standard Javascript is on doesn't mean it isn't an interpreter. All an interpreter is is something that executes a script directly rather than requiring compilation. It is a defacto interpreter for a subset of javascript. Nothing more, nothing less. The title could be more clear, however.

baobabKoodaa 4y ago
There's a huge difference between an interpreter for "JavaScript" and an interpreter for a "subset of JavaScript".
- Test0129 4y ago
  Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter. I'll continue to eat downvotes on this because of the pedantry of HN.
  - khazhoux 4y ago
    Technically, it’s only the pedantry of a subset of HN.
    - lupire 4y ago
      It's an interpretation of a subset of the pedantry on HN.
  - jraph 4y ago
    I didn't downvote, but I don't think esprehn is being unfair. Their comment is very informative. They didn't argue that what was implemented is not an interpreter, they did explain why it's not a JavaScript interpreter and not even an interpreter for a subset of JavaScript. It's just a special purpose interpreter suitable for YouTube's code that cannot be re-used for any code that uses the subset that it seems to implement. It's not pedantry (or I'm pedantic). It's a reaction to the title that can lead people to believe that a complete JavaScript interpreter has been written in less than a thousand lines of Python. This reaction is perfectly understandable.
  - chess_buster 4y ago
    I evaluated it with my Pedantic Interpreter which only results in the `pedantic` token.
  - baobabKoodaa 4y ago
    > Making a pedantic argument on what constitutes an interpreter is silly. The title is bad. It is an interpreter. It's not a pedantic argument. Based on the title I thought that somebody wrote something akin to V8 in 800 lines of Python. After reading the comments I realized those 800 lines just interpret a particular JavaScript function written by Youtube. Those things are different. Pointing out the fact that they are different is not pedantry. The title is misleading and the comments pointing that out are helpful.
  - blondin 4y ago
    my vote is meaningless and i am sorry about that. but just wanted to let you know that what you said made sense. do not let people get to you. most of us know that a thousand or so lines of code is not a full JavaScript interpreter and cannot be the real thing. there is no argument or conversation to have about it.
deleted 4y ago
[deleted]
blast 4y ago
esprehn didn't say it isn't an interpreter. They're saying it is an interpreter and what it's interpreting isn't (all of) JS. That's also what you're saying, so you're agreeing with esprehn. Edit: You misunderstood baobabKoodaa in the same way. Nobody is arguing about what constitutes an interpreter, except you. The question is what language is being interpreted. Before accusing someone of pedantry, it would first be good not to completely misread them.

jraph 4y ago

And as a user of youtube-dl, I'm quite happy about this. This probably allows a very safe, restricted "subset" of JS. Way better than using a full JS engine. 900 lines is still small and manageable.

sebzim4500 4y ago
I'm trying to get the thread model here. Is the concern that Youtube will inject JS into the payload which tries to break out of the youtuble-dl js sandbox using some zero day in whatever js engine they would use instead?
- kevingadd 4y ago
  Embedding a whole js engine and then interopping with it from python would be non trivial. Good luck fixing any bugs or corner cases you hit that way. The V8 and spidermonkey embedding apis are both c++ (iirc) and non trivial to use correctly. Having full control like this +simple code is probably lower risk and more maintainable, even if there's the challenge of expanding feature set if scripts change. The alternative would be a console js shell, but those are very different from browsers so that poses it's own challenges.
  - esprehn 4y ago
    Fwiw there are python bindings for QuickJS and Duktape: https://github.com/PetterS/quickjs https://github.com/PetterS/quickjs https://github.com/stefano/pyduktape https://github.com/stefano/pyduktape https://github.com/amol-/dukpy https://github.com/amol-/dukpy I can't speak to the quality of those bindings, but they do seem maintained.
  - em-bee 4y ago
    apparently yt-dlp is somehow calling out to a js engine if available
    - kevingadd 4y ago
      Yeah, it's possible to install v8 or spidermonkey shells and use them to run code - we use them to run parts of the .NET wasm test suite - but they have a bunch of arbitrary limitations, so if you're trying to emulate a browser I'm not sure I'd bet on them. It's certainly going to be easier than a C++ embedding, so it makes sense that they took that route. Another option is to use node, but it also has weird limitations/behaviors when running code.

blast 4y ago

I suppose this means it would be easy for YouTube to fuck with youtube-dl simply by throwing in more features of JS?

_pn3l 4y ago
Cat, meet mouse.
- nyanpasu64 4y ago
  It's unfortunate, https://github.com/mpv-player/mpv/issues/8655#issuecomment-1199733728 https://github.com/mpv-player/mpv/issues/8655#issuecomment-1...: > Youtube now throttles requests of more than 10MB at a time, yt-dlp works around it by making many requests of 10MB using Range HTTP headers (yt-dlp calls it the http-chunk-size), but ffmpeg which does the downloading for mpv doesn't support that yet. I want to change mpv or yt-dlp to support range-based video URLs (eg. appending &range=333999644-335298975&rn=5&rbuf=0 to URLs) which speed up stream seeking and probably eliminate throttling altogether, but I haven't taken the time to look into how to achieve it. For anyone interested, I have an open bug report at https://github.com/mpv-player/mpv/issues/10601 https://github.com/mpv-player/mpv/issues/10601, and have found https://satadalsengupta.github.io/docs/papers/2017_nossdav_youtubedash.pdf https://satadalsengupta.github.io/docs/papers/2017_nossdav_y... describing these parameters.

dang 4y ago

Ok, we've changed this title to shrink the scope of the interpreter. Submitted title was "YouTube-dl has a JavaScript interpreter written in 870 lines of Python".

ec109685 4y ago
Hence why HN better than Twitter. The amount of high engagement just plain wrong tweets there are is just sad.

deleted 4y ago

[deleted]