10 ms·

Reverse engineering the obfuscated TikTok VM

412 points by xfeeefeee 1y ago

xfeeefeee 1y ago
The fascinating process of reverse engineering this VM is detailed here. TikTok uses a custom virtual machine (VM) as part of its obfuscation and security layers. This project includes tools to: Deobfuscate webmssdk.js that has the virtual machine. Decompile TikTok’s virtual machine instructions into readable form. Script Inject Replace webmssdk.js with the deobfuscated VM injector. Sign URLs Generate signed URLs which can be used to perform auth-based requests eg. Post comments.
- noduerme 1y ago
  Is calling a massive embedded JS obfuscator a "VM" a bit of a stretch? Ultimately it's not translating anything to a lower-level language. Still, I had no idea. This is really taking JS obfuscation to the next level. One kind of wonders, what is the purpose of that level of obfuscation? The naive take is that obfuscation is usually to protect intellectual property... but this is client-side code that wouldn't give away anything about their secret sauce algorithm.
  - throwaway48476 1y ago
    VM obfuscation is a common technique for malware developers. The VM term is applied because the obfuscator creates a custom instruction set and executes custom byte code. This is generated per build.
    - noduerme 1y ago
      I appreciate you making the distinction that anything which creates a custom instruction set is thus a VM. I think that's the way a lot of people here who are currently at my throat seem to define it, so I'm glad you put it in clear terms. I would define it as a custom instruction set plus some sort of plug-in that allows those opcodes to be run closer to the metal than the language they're written in. FWIW I'd call this thing more of an obfuscation framework. But maybe I'm just a dino. I am really glad you made this comment, though. It clarified for me why so many people went bananas when I said this wasn't a VM.
  - MonkeyClub 1y ago
    > Is calling a massive embedded JS obfuscator a "VM" a bit of a stretch? Ultimately it's not translating anything to a lower-level language. From the Repo's README: "TikTok is using a full-fledged bytecode VM, if you browse through it, it supports scopes, nested functions and exception handling. This isn't a typical VM and shows that it is definitely sophiscated."
    - noduerme 1y ago
      But that's basically an emulator of a VM, isn't it? It's like rewriting the Flash AVM2 into JS... it's still running in JS whereas the original VM was C++. It could JIT compile stuff but only because it literally was reserving memory that could overflow, and (semi-technical take here) from that advantage, of being closer to the metal, flowed all of the flaws in AVM2 that precipitated most of Adobe's woes with Flash. A VM implant in a web page that uses a plugin like Java or Flash, to get around running browser-sandboxed code, which can take over physical memory, is far different from just emulating a VM in Javascript. I wouldn't call writing a ton of opcodes in JS, which resolved to JS functions, a "virtual machine", because it isn't reserving anything or doing anything that Javascript can't do. Someone correct me here if I'm wrong... this is just heavy-duty obfuscation. Also, one major purpose of a VM is to improve performance over what's available in the browser. If you use that as a measurement, this clearly doesn't fit that goal.
      - deleted 1y ago
        [deleted]
      - gruez 1y ago
        >But that's basically an emulator of a VM, isn't it? Emulators and VMs aren't mutually exclusive. >Also, one major purpose of a VM is to improve performance over what's available in the browser. If you use that as a measurement, this clearly doesn't fit that goal. And from your other comment: >I would define it as a custom instruction set plus some sort of plug-in that allows those opcodes to be run closer to the metal than the language they're written in. A virtual machine just means a machine that's virtual. All the other expectations you apply on top of it (eg. "improve performance over what's available in the browser") is totally irrelevant. The JVM clearly doesn't improve performance of java code than running natively, but nobody denies it's a virtual machine. The same goes for VMWare products ("VM" is literally in its name!), which executes x86 code but is further away from "the metal" that it's running on.
  - userbinator 1y ago
    You are replying to a comment that looks extremely unhuman.
    - codetrotter 1y ago
      It looks like OP filled out the text area alongside with the URL when submitting the post. HN takes that text and turns it into a comment. I’ve seen it happen before. The unfortunate outcome of that IMO is that sometimes text that makes sense as a description of a submission feels a bit out of place as a comment due to how they are worded. And these comments sometimes then end up getting downvoted. I wouldn’t be completely sure it was not human written. Even though it feels a bit weird to read it as a comment.
      - lukaso112 1y ago
        [dead]
      - xfeeefeee 1y ago
        > It looks like OP filled out the text area alongside with the URL when submitting the post. HN takes that text and turns it into a comment. Yeah, this is exactly what happened, but I decided to keep it rather than delete and filled it out more with the synopsis from the repo. Looking back at it, it really does look like an AI bulleted summary. I probably should have noted that the last part was indeed a quotation.
- dmitrygr 1y ago
  What is the purpose of you posting a bad ChatGPT summary of the original post?
  - xfeeefeee 1y ago
    I quoted the synopsis from the readme thinking it would be helpful.
  - pests 1y ago
    It was the submission statement along with him submitting this post. It was detached as a comment and I don't think it AI.
godelski 1y ago
This seems like quite a lot of work to hide the code. What would the legitimate reasons for this be? Because it looks like it would make the program less optimized and more complexity just leads to more errors. I understand the desire to make it harder for bots, but 1) it doesn't seem to be effective and bots seem to be going a very different route 2) there's got to be better ways that are more effective. It's not like you're going to stop clones through this because clones can replicate by just seeing how things work and reverse engineer blackbox style.
- davidsojevic 1y ago
  Making it harder for bots usually means that it drives up the cost for the bots to operate; so if they need to run in a headless browser to get around the anti-bot measures it might mean that it takes, for example, 1.5 seconds to execute a request as compared to the 0.1 seconds it would without them in place. On top of that 1.5 seconds is also that there is a much larger CPU and memory cost from having to run that browser compared to a simple direct HTTP request which is near negligible. So while you'll never truly defeat a sufficiently motivated actor, you may be able to drive their costs up high enough that it makes it difficult to enter the space or difficult to turn a profit if they're so inclined.
  - godelski 1y ago
    I understand the argument. You can't have perfect defense and speedbumps are quite effective. I'm not trying to disagree with that. But it does not seem like the solution is effective at mitigating bots. Presumably bots are going a different route considering how prolific they are, which warrants another solution. If they are going through this route then it certainly isn't effective either and also warrants another solution. It seems like this obscurification requires a fair amount of work, especially since you need to frequently update the code to rescramble it. Added complexity also increases risks for bugs and vulnerabilities, which ultimately undermine the whole endeavor. I'm trying to understand why this level of effort is worth the cost. (Other than nefarious reasons. Those ones are rather obvious)
- noduerme 1y ago
ronsor 1y ago
There is no legitimate reason for a social media platform to employ this much obfuscation.
- miohtama 1y ago
  It's to keep bots away and not turn to be another Twitter.
  - dns_snek 1y ago
    That's probably not the goal. There are bots advertising illegal services (e.g. ads for "hacking services", illegal drugs) in most comment sections. If you report these comments, 99.9% of the time the report will be rejected with "no violations found" and the spam stays up.
    - bolognafairy 1y ago
      That doesn’t mean that it’s “probably not the intention”.
      - dns_snek 1y ago
        The balance of evidence suggests otherwise. If they cared about spam bots they would take action when spammers are handed to them on a silver platter. The kinds of spammers who will leave 30 identical comments advertising illegal services, not some weird moderation corner case. If you ever end up on a video that's related to drugs, there will be entire chains of bots just advertising to each other and TikTok won't find any violations when reported. But sure, I'm sure they care a whole lot about not ending up like Twitter.
        TheDong 1y ago
        So you're saying that TikTok's support team doing a poor job of handling reports is proof that the engineering team wasn't tasked with reducing spam by writing code obfuscation? TikTok is a huge company, evidence of what the support department does or doesn't do has only minor bearing on the whole company, and basically none on the engineering department. The thing that seems most likely to me is that they care about spam, the engineering department did this one thing, and the support department is either overworked or cares less. Or really efficient which is why you only see "a lot of spam", not "literally nothing but spam".
davidsojevic 1y ago
Very impressive work! I always enjoy a good write up about reverse engineering efforts and yours was really simple to follow. Many popular/large websites and bot protection services usually have environment checking as a baseline and mouse-movement tracking in some of the more aggressive anti-bot checks. It's always interesting to see how long it takes from when the measures have been defeated/publicised until the service ends up making changes to their mechanism to make you start over (hopefully not from scratch).
- xfeeefeee 1y ago
  All credit should go to Lukas https://github.com/LukasOgunfeitimi https://github.com/LukasOgunfeitimi I was sharing this here since I thought it was a great write up, but did not intend to pass it off as my own! There is certainly always a good amount of push and pull, though my personal concern as a contributor to yt-dlp under another alias is more about archival of the underlying media rather than automating things like comments. YouTube also uses an interesting scheme for authenticating requests for media as well which required implementing a very basic JavaScript interpreter within Python for yt-dlp too. I expect this kind of thing to continue to become even more common and complicated.
worldsavior 1y ago
That's a very strong obfuscation. Takes a lot of work to deobfuscate such a thing. Great writeup.
domfie 1y ago
Looks like a lot of work. I recently discovered webcrack and the tool jehna/humanify for such deobfuscate tasks
- 3abiton 1y ago
  It could be interesting to see a comparison to OP's work.
0xDEADFED5 1y ago
this is cool. i briefly worked on a TikTok bot a while back and it was a huge pain in the ass.
heinternets 1y ago
Is TikTok so obfuscated to prevent people from knowing the full extent of data collection and device fingerprinting?
- gruez 1y ago
  1. Practically speaking all this javascript fingerprinting pales in comparison to what native apps have access to. Most people aren't using tiktok on their browsers, and the browser version heavily pushes you to using the app, so you should be far more worried about whatever's happening in the app. 2. Despite tiktok having a giant target painted on its back for its perceived connections to the CCP, I haven't really seen any evidence that it does any more tracking/fingerprinting that most other websites (eg. facebook) or security services (eg. cloudflare or recaptcha) already do.
  - nicce 1y ago
    > 2. Despite tiktok having a giant target painted on its back for its perceived connections to the CCP, I haven't really seen any evidence that it does any more tracking/fingerprinting that most other websites (eg. facebook) or security services (eg. cloudflare or recaptcha) already do. Take a look for request parameters in TikTok vs. Instagram for example. Every request for TikTok forces you to pass most of the information that browser can collect from the end-user before server responds: https://www.nullpt.rs/reverse-engineering-tiktok-vm-1 https://www.nullpt.rs/reverse-engineering-tiktok-vm-1
    - gruez 1y ago
      >Every request for TikTok forces you to pass most of the information that browser can collect from the end-user before server responds: Half of the parameters are stuff relating to the app itself, or could be inferred from other sources like user-agent. The other fingerprinting stuff (eg. canvas or webgl fingerprinting) is basically industry standard and by no means unique to tiktok. Even the claim that "browser can collect from the end-user before server responds" doesn't hold up to scrutiny, because there's no meaningful difference between that, and browser check interstitials (eg. the cloudflare checkbox), which fingerprint you before letting you access the content. It's also unclear how that's more sinister than the alternative approach of sending telemetry/fingerprinting data to a separate endpoint.
RexM 1y ago
Is this VM somehow related to Lynx (their cross platform dev tooling?) https://lynxjs.org/ https://lynxjs.org/ Also discussed on HN https://news.ycombinator.com/item?id=43264957 https://news.ycombinator.com/item?id=43264957
weinzierl 1y ago
Is there also a VM in their iOS app? I thought a VM would be against Apple's policies?
- xmodem 1y ago
  Apple's policies prevent using JIT compilation, they don't ban VM's outright.
  - jacobp100 1y ago
    This is the correct answer. They even expose JavaScript Core to apps
- Scaevolus 1y ago
  Their mobile apps have equivalent signature code, but it's compiled to native binaries instead.
kleiba 1y ago
I've been using a shitty streaming website whose player interrupts the playback of a video in irregular intervals and presents a cryptic error message. I've started looking into the JavaScript code to see if I can't code up a work-around mechanism (basically debugging their garbage implementation), and of course (why actually?) their player code is also obfuscated. And I've gotta say, emplying an AI assistant has proven to be an invaluable help in trying to understand obfuscated code. It's actually really cool to take a function of gobbledegook JavaScript and ask the AI to rewrite it in a more canonical and easily understandable way, with inline comments. Of course, there are flaws every now and then, but the ability to do this has been such a game changer for reverse engineering, IMO. I can even ask to take a guess at finding better variable/function names and the AI can infer from the code (maybe has seen the unobfuscated libraries during training?) what this code is actually doing on a high-level and turn something like e.g(e.g) into player.initialize(player.state) which is nothing short of amazing. So for anyone doing similar work, I cannot recommend highly enough to have an AI agent as another tool in your tool belt.
- lukan 1y ago
  Which AI agents did you use?
  - kleiba 1y ago
    I've tried different ones, they all seem to do a great job.
    - klabetron 1y ago
      Out of curiosity (as someone disappointingly new to prompt engineering), what’s an example prompt you used with some success?
      - esseph 1y ago
        Ask questions. Be disappointed in the outcomes. Ask more questions. Get some right answers. Repeat. Make question asking muscle get swole.
sylware 1y ago
What's terrible are the humans writing such software... But if AI can help to fight those people's work, good for humanity I guess. That said... Is AI going to de-obfuscate/reverse engineer their obsfuscated AI prompts or web apps?
SoKamil 1y ago
> As this is a Javascript file executed on the web, it is actually possible to replace the normal webmssdk.js with the deobfuscated file and use TikTok normally. > This can be achieved by using two browser extensions known as Tampermonkey for executing custom code and CSP to disable CSP so I can fetch files from blocked origins. This is so I can put latestDeobf.js in my own file server and have it be fetched each time, this is so I can easily edit the file and let the changes take effect each time I refresh. This makes it much easier to bebug when reversing functions. I believe you can achieve the same effect without any 3rd party extensions. You can use Local Overrides in Chrome DevTools. Great work!
- wutwutwat 1y ago
  You can also install some trusted certs and MITM the requests, replacing the content with whatever you'd like Likely overkill for this use case, but no matter the client, you can in theory do whatever you want to any traffic up until the point it leaves your network.
  - ImPostingOnHN 1y ago
    what toolset do you use for on-the-fly translation? ad-hoc code, or something with a more structured workflow, maybe? this sounds like a fun thing to try, thanks for your time
    - 18172828286177 1y ago
      See Burpsuite
    - SoKamil 1y ago
      Charles, Proxyman, or mitmproxy if you like open source + terminal would do the job.
      - geoka9 1y ago
        mitmproxy will even allow you to script the intercept/override behavior, which can be really handy.
Wowfunhappy 1y ago
...can I ask a really stupid question? What is a VM in this context? I've used VM's for years to run Windows on top of macOS or Linux on top of Windows or macOS on top of macOS when I need an isolated testing environment. I also know that Java works via the "Javascript Virtual Machine" which I've always thought of as "Java code actually runs in its own lightweight operating system on top of the host OS, which makes it OS-agnostic". The JVM can't run on bare metal because it doesn't have hardware drivers, but presumably it could if you wrote those drivers. But presumably the VM being discussed in TFA isn't that kind of VM, right? Bytedance didn't write an operating system in Javascript? I've been seeing "VM" used in lots of contexts like this recently and it makes me think I must be missing something, but it's the sort of question I don't know how to Google. AIs have not been helpful either, plus I don't trust them.
- jacobp100 1y ago
  Yes the VM discussed is similar to JVM
- turtleyacht 1y ago
  Virtual Machine Decompiling: https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineering?tab=readme-ov-file#virtual-machine-decompiling https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineerin... And also VM223, with statements that do stuff to an array "stack": https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineering/blob/main/decompiler/functions/VM223.js https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineerin... One obvious giveaway for a VM is laying out memory, or processing some intermediate language. In this case, it could be the latter. In-browser, you have Chrome V8 running Javascript; that Javascript could be running an interpreted environment where abstractions are not purely business logic, but an execution model separate from domain stuff: auth, video, user, etc. By that observation, this C snippet is a VM: char instruction = 'p'; /* or array */ if (instruction == 'p') { println("document.appendChild(...)"); } If the program outputs to a vm.js file, it's kinda-sorta a "VM." I would call it something else, maybe a generator of sorts (for now). Just in my opinion, for me, if I were working on a VM, the threshold of calling it that would be much higher than the above. On the other hand, if I had to comment in the generated Javascript debugging hints referring to execution stack or stack pointers, it is kind of a VM idea.
lukaso112 1y ago
[dead]
itsthecourier 1y ago
this level of obfuscation in a social app is super suspicious
- doublerabbit 1y ago
  I wouldn't say so, pretty common. It used to add a layer of security. You should take a look at an casino app. Did you know that every chip on a Chip & Pin bank card is powered by a Java Virtual Machine that when you go to tap or insert in to a card reader it's activated. https://en.wikipedia.org/wiki/Java_Card https://en.wikipedia.org/wiki/Java_Card
mrkramer 1y ago
In my bookmarks I found this RE examples as well: https://www.nullpt.rs/reverse-engineering-tiktok-vm-1 https://www.nullpt.rs/reverse-engineering-tiktok-vm-1 https://ibiyemiabiodun.com/projects/reversing-tiktok-pt2/ https://ibiyemiabiodun.com/projects/reversing-tiktok-pt2/
lazyeye 1y ago
An oldie but a goodie. A guide to manipulating online comments to hide/dilute/obsfucate undesirable commentary.... https://cryptome.org/2012/07/gent-forum-spies.htm https://cryptome.org/2012/07/gent-forum-spies.htm
deleted 1y ago
[deleted]

412 points by xfeeefeee 1y ago

The fascinating process of reverse engineering this VM is detailed here. TikTok uses a custom virtual machine (VM) as part of its obfuscation and security layers. This project includes tools to: Deobfuscate webmssdk.js that has the virtual machine. Decompile TikTok’s virtual machine instructions into readable form. Script Inject Replace webmssdk.js with the deobfuscated VM injector. Sign URLs Generate signed URLs which can be used to perform auth-based requests eg. Post comments.

noduerme 1y ago
Is calling a massive embedded JS obfuscator a "VM" a bit of a stretch? Ultimately it's not translating anything to a lower-level language. Still, I had no idea. This is really taking JS obfuscation to the next level. One kind of wonders, what is the purpose of that level of obfuscation? The naive take is that obfuscation is usually to protect intellectual property... but this is client-side code that wouldn't give away anything about their secret sauce algorithm.
- throwaway48476 1y ago
  VM obfuscation is a common technique for malware developers. The VM term is applied because the obfuscator creates a custom instruction set and executes custom byte code. This is generated per build.
  - noduerme 1y ago
    I appreciate you making the distinction that anything which creates a custom instruction set is thus a VM. I think that's the way a lot of people here who are currently at my throat seem to define it, so I'm glad you put it in clear terms. I would define it as a custom instruction set plus some sort of plug-in that allows those opcodes to be run closer to the metal than the language they're written in. FWIW I'd call this thing more of an obfuscation framework. But maybe I'm just a dino. I am really glad you made this comment, though. It clarified for me why so many people went bananas when I said this wasn't a VM.
- MonkeyClub 1y ago
  > Is calling a massive embedded JS obfuscator a "VM" a bit of a stretch? Ultimately it's not translating anything to a lower-level language. From the Repo's README: "TikTok is using a full-fledged bytecode VM, if you browse through it, it supports scopes, nested functions and exception handling. This isn't a typical VM and shows that it is definitely sophiscated."
  - noduerme 1y ago
    But that's basically an emulator of a VM, isn't it? It's like rewriting the Flash AVM2 into JS... it's still running in JS whereas the original VM was C++. It could JIT compile stuff but only because it literally was reserving memory that could overflow, and (semi-technical take here) from that advantage, of being closer to the metal, flowed all of the flaws in AVM2 that precipitated most of Adobe's woes with Flash. A VM implant in a web page that uses a plugin like Java or Flash, to get around running browser-sandboxed code, which can take over physical memory, is far different from just emulating a VM in Javascript. I wouldn't call writing a ton of opcodes in JS, which resolved to JS functions, a "virtual machine", because it isn't reserving anything or doing anything that Javascript can't do. Someone correct me here if I'm wrong... this is just heavy-duty obfuscation. Also, one major purpose of a VM is to improve performance over what's available in the browser. If you use that as a measurement, this clearly doesn't fit that goal.
    - deleted 1y ago
      [deleted]
    - gruez 1y ago
      >But that's basically an emulator of a VM, isn't it? Emulators and VMs aren't mutually exclusive. >Also, one major purpose of a VM is to improve performance over what's available in the browser. If you use that as a measurement, this clearly doesn't fit that goal. And from your other comment: >I would define it as a custom instruction set plus some sort of plug-in that allows those opcodes to be run closer to the metal than the language they're written in. A virtual machine just means a machine that's virtual. All the other expectations you apply on top of it (eg. "improve performance over what's available in the browser") is totally irrelevant. The JVM clearly doesn't improve performance of java code than running natively, but nobody denies it's a virtual machine. The same goes for VMWare products ("VM" is literally in its name!), which executes x86 code but is further away from "the metal" that it's running on.
- userbinator 1y ago
  You are replying to a comment that looks extremely unhuman.
  - codetrotter 1y ago
    It looks like OP filled out the text area alongside with the URL when submitting the post. HN takes that text and turns it into a comment. I’ve seen it happen before. The unfortunate outcome of that IMO is that sometimes text that makes sense as a description of a submission feels a bit out of place as a comment due to how they are worded. And these comments sometimes then end up getting downvoted. I wouldn’t be completely sure it was not human written. Even though it feels a bit weird to read it as a comment.
    - lukaso112 1y ago
      [dead]
    - xfeeefeee 1y ago
      > It looks like OP filled out the text area alongside with the URL when submitting the post. HN takes that text and turns it into a comment. Yeah, this is exactly what happened, but I decided to keep it rather than delete and filled it out more with the synopsis from the repo. Looking back at it, it really does look like an AI bulleted summary. I probably should have noted that the last part was indeed a quotation.
dmitrygr 1y ago
What is the purpose of you posting a bad ChatGPT summary of the original post?
- xfeeefeee 1y ago
  I quoted the synopsis from the readme thinking it would be helpful.
- pests 1y ago
  It was the submission statement along with him submitting this post. It was detached as a comment and I don't think it AI.

godelski 1y ago

This seems like quite a lot of work to hide the code. What would the legitimate reasons for this be? Because it looks like it would make the program less optimized and more complexity just leads to more errors. I understand the desire to make it harder for bots, but 1) it doesn't seem to be effective and bots seem to be going a very different route 2) there's got to be better ways that are more effective. It's not like you're going to stop clones through this because clones can replicate by just seeing how things work and reverse engineer blackbox style.

davidsojevic 1y ago
Making it harder for bots usually means that it drives up the cost for the bots to operate; so if they need to run in a headless browser to get around the anti-bot measures it might mean that it takes, for example, 1.5 seconds to execute a request as compared to the 0.1 seconds it would without them in place. On top of that 1.5 seconds is also that there is a much larger CPU and memory cost from having to run that browser compared to a simple direct HTTP request which is near negligible. So while you'll never truly defeat a sufficiently motivated actor, you may be able to drive their costs up high enough that it makes it difficult to enter the space or difficult to turn a profit if they're so inclined.
- godelski 1y ago
  I understand the argument. You can't have perfect defense and speedbumps are quite effective. I'm not trying to disagree with that. But it does not seem like the solution is effective at mitigating bots. Presumably bots are going a different route considering how prolific they are, which warrants another solution. If they are going through this route then it certainly isn't effective either and also warrants another solution. It seems like this obscurification requires a fair amount of work, especially since you need to frequently update the code to rescramble it. Added complexity also increases risks for bugs and vulnerabilities, which ultimately undermine the whole endeavor. I'm trying to understand why this level of effort is worth the cost. (Other than nefarious reasons. Those ones are rather obvious)
noduerme 1y ago

ronsor 1y ago

There is no legitimate reason for a social media platform to employ this much obfuscation.

miohtama 1y ago
It's to keep bots away and not turn to be another Twitter.
- dns_snek 1y ago
  That's probably not the goal. There are bots advertising illegal services (e.g. ads for "hacking services", illegal drugs) in most comment sections. If you report these comments, 99.9% of the time the report will be rejected with "no violations found" and the spam stays up.
  - bolognafairy 1y ago
    That doesn’t mean that it’s “probably not the intention”.
    - dns_snek 1y ago
      The balance of evidence suggests otherwise. If they cared about spam bots they would take action when spammers are handed to them on a silver platter. The kinds of spammers who will leave 30 identical comments advertising illegal services, not some weird moderation corner case. If you ever end up on a video that's related to drugs, there will be entire chains of bots just advertising to each other and TikTok won't find any violations when reported. But sure, I'm sure they care a whole lot about not ending up like Twitter.
      - TheDong 1y ago
        So you're saying that TikTok's support team doing a poor job of handling reports is proof that the engineering team wasn't tasked with reducing spam by writing code obfuscation? TikTok is a huge company, evidence of what the support department does or doesn't do has only minor bearing on the whole company, and basically none on the engineering department. The thing that seems most likely to me is that they care about spam, the engineering department did this one thing, and the support department is either overworked or cares less. Or really efficient which is why you only see "a lot of spam", not "literally nothing but spam".

davidsojevic 1y ago

Very impressive work! I always enjoy a good write up about reverse engineering efforts and yours was really simple to follow. Many popular/large websites and bot protection services usually have environment checking as a baseline and mouse-movement tracking in some of the more aggressive anti-bot checks. It's always interesting to see how long it takes from when the measures have been defeated/publicised until the service ends up making changes to their mechanism to make you start over (hopefully not from scratch).

xfeeefeee 1y ago
All credit should go to Lukas https://github.com/LukasOgunfeitimi https://github.com/LukasOgunfeitimi I was sharing this here since I thought it was a great write up, but did not intend to pass it off as my own! There is certainly always a good amount of push and pull, though my personal concern as a contributor to yt-dlp under another alias is more about archival of the underlying media rather than automating things like comments. YouTube also uses an interesting scheme for authenticating requests for media as well which required implementing a very basic JavaScript interpreter within Python for yt-dlp too. I expect this kind of thing to continue to become even more common and complicated.

worldsavior 1y ago

That's a very strong obfuscation. Takes a lot of work to deobfuscate such a thing. Great writeup.

domfie 1y ago

Looks like a lot of work. I recently discovered webcrack and the tool jehna/humanify for such deobfuscate tasks

3abiton 1y ago
It could be interesting to see a comparison to OP's work.

0xDEADFED5 1y ago

this is cool. i briefly worked on a TikTok bot a while back and it was a huge pain in the ass.

heinternets 1y ago

Is TikTok so obfuscated to prevent people from knowing the full extent of data collection and device fingerprinting?

gruez 1y ago
1. Practically speaking all this javascript fingerprinting pales in comparison to what native apps have access to. Most people aren't using tiktok on their browsers, and the browser version heavily pushes you to using the app, so you should be far more worried about whatever's happening in the app. 2. Despite tiktok having a giant target painted on its back for its perceived connections to the CCP, I haven't really seen any evidence that it does any more tracking/fingerprinting that most other websites (eg. facebook) or security services (eg. cloudflare or recaptcha) already do.
- nicce 1y ago
  > 2. Despite tiktok having a giant target painted on its back for its perceived connections to the CCP, I haven't really seen any evidence that it does any more tracking/fingerprinting that most other websites (eg. facebook) or security services (eg. cloudflare or recaptcha) already do. Take a look for request parameters in TikTok vs. Instagram for example. Every request for TikTok forces you to pass most of the information that browser can collect from the end-user before server responds: https://www.nullpt.rs/reverse-engineering-tiktok-vm-1 https://www.nullpt.rs/reverse-engineering-tiktok-vm-1
  - gruez 1y ago
    >Every request for TikTok forces you to pass most of the information that browser can collect from the end-user before server responds: Half of the parameters are stuff relating to the app itself, or could be inferred from other sources like user-agent. The other fingerprinting stuff (eg. canvas or webgl fingerprinting) is basically industry standard and by no means unique to tiktok. Even the claim that "browser can collect from the end-user before server responds" doesn't hold up to scrutiny, because there's no meaningful difference between that, and browser check interstitials (eg. the cloudflare checkbox), which fingerprint you before letting you access the content. It's also unclear how that's more sinister than the alternative approach of sending telemetry/fingerprinting data to a separate endpoint.

RexM 1y ago

Is this VM somehow related to Lynx (their cross platform dev tooling?) https://lynxjs.org/ https://lynxjs.org/ Also discussed on HN https://news.ycombinator.com/item?id=43264957 https://news.ycombinator.com/item?id=43264957

weinzierl 1y ago

Is there also a VM in their iOS app? I thought a VM would be against Apple's policies?

xmodem 1y ago
Apple's policies prevent using JIT compilation, they don't ban VM's outright.
- jacobp100 1y ago
  This is the correct answer. They even expose JavaScript Core to apps
Scaevolus 1y ago
Their mobile apps have equivalent signature code, but it's compiled to native binaries instead.

kleiba 1y ago

I've been using a shitty streaming website whose player interrupts the playback of a video in irregular intervals and presents a cryptic error message. I've started looking into the JavaScript code to see if I can't code up a work-around mechanism (basically debugging their garbage implementation), and of course (why actually?) their player code is also obfuscated. And I've gotta say, emplying an AI assistant has proven to be an invaluable help in trying to understand obfuscated code. It's actually really cool to take a function of gobbledegook JavaScript and ask the AI to rewrite it in a more canonical and easily understandable way, with inline comments. Of course, there are flaws every now and then, but the ability to do this has been such a game changer for reverse engineering, IMO. I can even ask to take a guess at finding better variable/function names and the AI can infer from the code (maybe has seen the unobfuscated libraries during training?) what this code is actually doing on a high-level and turn something like e.g(e.g) into player.initialize(player.state) which is nothing short of amazing. So for anyone doing similar work, I cannot recommend highly enough to have an AI agent as another tool in your tool belt.

lukan 1y ago
Which AI agents did you use?
- kleiba 1y ago
  I've tried different ones, they all seem to do a great job.
  - klabetron 1y ago
    Out of curiosity (as someone disappointingly new to prompt engineering), what’s an example prompt you used with some success?
    - esseph 1y ago
      Ask questions. Be disappointed in the outcomes. Ask more questions. Get some right answers. Repeat. Make question asking muscle get swole.

sylware 1y ago

What's terrible are the humans writing such software... But if AI can help to fight those people's work, good for humanity I guess. That said... Is AI going to de-obfuscate/reverse engineer their obsfuscated AI prompts or web apps?

SoKamil 1y ago

> As this is a Javascript file executed on the web, it is actually possible to replace the normal webmssdk.js with the deobfuscated file and use TikTok normally. > This can be achieved by using two browser extensions known as Tampermonkey for executing custom code and CSP to disable CSP so I can fetch files from blocked origins. This is so I can put latestDeobf.js in my own file server and have it be fetched each time, this is so I can easily edit the file and let the changes take effect each time I refresh. This makes it much easier to bebug when reversing functions. I believe you can achieve the same effect without any 3rd party extensions. You can use Local Overrides in Chrome DevTools. Great work!

wutwutwat 1y ago
You can also install some trusted certs and MITM the requests, replacing the content with whatever you'd like Likely overkill for this use case, but no matter the client, you can in theory do whatever you want to any traffic up until the point it leaves your network.
- ImPostingOnHN 1y ago
  what toolset do you use for on-the-fly translation? ad-hoc code, or something with a more structured workflow, maybe? this sounds like a fun thing to try, thanks for your time
  - 18172828286177 1y ago
    See Burpsuite
  - SoKamil 1y ago
    Charles, Proxyman, or mitmproxy if you like open source + terminal would do the job.
    - geoka9 1y ago
      mitmproxy will even allow you to script the intercept/override behavior, which can be really handy.

Wowfunhappy 1y ago

...can I ask a really stupid question? What is a VM in this context? I've used VM's for years to run Windows on top of macOS or Linux on top of Windows or macOS on top of macOS when I need an isolated testing environment. I also know that Java works via the "Javascript Virtual Machine" which I've always thought of as "Java code actually runs in its own lightweight operating system on top of the host OS, which makes it OS-agnostic". The JVM can't run on bare metal because it doesn't have hardware drivers, but presumably it could if you wrote those drivers. But presumably the VM being discussed in TFA isn't that kind of VM, right? Bytedance didn't write an operating system in Javascript? I've been seeing "VM" used in lots of contexts like this recently and it makes me think I must be missing something, but it's the sort of question I don't know how to Google. AIs have not been helpful either, plus I don't trust them.

jacobp100 1y ago
Yes the VM discussed is similar to JVM
turtleyacht 1y ago
Virtual Machine Decompiling: https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineering?tab=readme-ov-file#virtual-machine-decompiling https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineerin... And also VM223, with statements that do stuff to an array "stack": https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineering/blob/main/decompiler/functions/VM223.js https://github.com/LukasOgunfeitimi/TikTok-ReverseEngineerin... One obvious giveaway for a VM is laying out memory, or processing some intermediate language. In this case, it could be the latter. In-browser, you have Chrome V8 running Javascript; that Javascript could be running an interpreted environment where abstractions are not purely business logic, but an execution model separate from domain stuff: auth, video, user, etc. By that observation, this C snippet is a VM: char instruction = 'p'; /* or array */ if (instruction == 'p') { println("document.appendChild(...)"); } If the program outputs to a vm.js file, it's kinda-sorta a "VM." I would call it something else, maybe a generator of sorts (for now). Just in my opinion, for me, if I were working on a VM, the threshold of calling it that would be much higher than the above. On the other hand, if I had to comment in the generated Javascript debugging hints referring to execution stack or stack pointers, it is kind of a VM idea.

lukaso112 1y ago

[dead]

itsthecourier 1y ago

this level of obfuscation in a social app is super suspicious

doublerabbit 1y ago
I wouldn't say so, pretty common. It used to add a layer of security. You should take a look at an casino app. Did you know that every chip on a Chip & Pin bank card is powered by a Java Virtual Machine that when you go to tap or insert in to a card reader it's activated. https://en.wikipedia.org/wiki/Java_Card https://en.wikipedia.org/wiki/Java_Card

mrkramer 1y ago

In my bookmarks I found this RE examples as well: https://www.nullpt.rs/reverse-engineering-tiktok-vm-1 https://www.nullpt.rs/reverse-engineering-tiktok-vm-1 https://ibiyemiabiodun.com/projects/reversing-tiktok-pt2/ https://ibiyemiabiodun.com/projects/reversing-tiktok-pt2/

lazyeye 1y ago

An oldie but a goodie. A guide to manipulating online comments to hide/dilute/obsfucate undesirable commentary.... https://cryptome.org/2012/07/gent-forum-spies.htm https://cryptome.org/2012/07/gent-forum-spies.htm

deleted 1y ago

[deleted]