16 ms·

I had to go through a lot of comments in that thread to find a working workaround. On OS X, Open /Applications/Spotify.app/Contents/MacOS/Spotify in a hex edit

I had to go through a lot of comments in that thread to find a working workaround. On OS X, Open /Applications/Spotify.app/Contents/MacOS/Spotify in a hex editor. Search for "VACUUM;" Replace with "xxxxxx;" On Windows, apparently the key "VACUUM;" string is in libcef.dll but I don't have a Windows system to see if editing it the same way provides a workaround like on OS X. I used hexcurse from homebrew. After opening the file hit tab once to switch to the ASCII side, Control-F to search for VACUUM;, then type "x" six times to overwrite each character. Quit and save. EDIT: fixes

ihuman 10y ago
6 or 7 x's? In your example, you used 6.
- philipkglass 10y ago
  7 x's, one to overwrite all of VACUUM; including the terminating semicolon. I didn't count the x's in the quoted text that I copied from the thread.
  - 0x0 10y ago
    If you overwrite the ; is there not a risk you might cause the next legitimate sqlite operation to fail?
    - SturgeonsLaw 10y ago
      They might get lucky and overwrite the last SQL statement in the method. Seems likely with a statement like VACUUM.
- cjg_ 10y ago
  6, it wont start otherwise. I left the ; in.
  - philipkglass 10y ago
    Hmm, I used 7 and my patched client is running fine. Maybe it's enough to garble VACUUM so it doesn't do anything valid.
    - cjg_ 10y ago
      If you replaced the ; also, you also didn't change the total length of the strings, I think that's what the problem is.
elsurudo 10y ago
An explanation of what this does? Also, I'm assuming this has to be done after every update, and since Spotify auto-updates, not so convenient...
- philipkglass 10y ago
  I didn't read all of the original thread but apparently Spotify is overusing sqlite VACUUM, causing a modest amount of data to turn into a huge write volume. https://sqlite.org/lang_vacuum.html https://sqlite.org/lang_vacuum.html
- peeters 10y ago
  I'm more familiar with Postgres, but it basically instructs the table storage provider to clean up unused disk. Think of it roughly as defragmentation. You're trading work (in the form of writes and CPU) for reclaimed space. Unless SQLite has some semantics around it, replacing it with "xxxxxx;" will likely just cause it to run a bad command...might be more correct to replace it with something like "------;" (comment characters).
  - boulos 10y ago
    I was going to say "SELECT 1;" but it's two characters too many. Will all comment characters evaluate to whatever VACUUM does?
- cosmie 10y ago
  It likely just stops Spotify from VACUUM'ing the SQLite database it has. Spotify stores a local SQLite database file (by default on a Mac it's at ~/Library/Application Support/Spotify/PersistentCache/mercury.db ) The file is only about ~100MB (on my computer). With databases, you issue a vacuum command to defragment a database (reclaim space from deleted/updated rows, re-sort the data, etc). SQLite's VACUUM behavior basically just recreates the entire database from scratch in a temp file, then replaces the live file with that[1]. It makes the file smaller and more efficient to run queries against. Spotify must be triggering a VACUUM statement too aggressively (possibly after every change), rather than on a periodic schedule or after a certain amount of fragmentation. By breaking the VACUUM statement with this solution, you prevent the database from getting recreated so frequently. The file itself will grow slowly (since you're not reclaiming free space like a VACUUM would) and possibly slow down queries against the database, but that sounds like a far better tradeoff than excessive wear on the drive from recreating the file so often. [1]https://sqlite.org/lang_vacuum.html https://sqlite.org/lang_vacuum.html
Etheryte 10y ago
Use <Control-F> to search and <Control-S> to save, it seems you mixed them up in your otherwise very helpful comment. What's the source of the problem and the fix, though?
- philipkglass 10y ago
  Yep, mixed them up. See other comment about sqlite.
adrinavarro 10y ago
This fixes it, but it's likely any future update kills this temp fix. Hopefully Spotify will take care of this. I also noticed the extremely high writes less than a week ago. Wonder how long this has been going on.
mikeash 10y ago
Spotify updates itself fairly regularly, so you may need to reapply this fix from time to time.
- philipkglass 10y ago
  Brute-force solution: save this as spotify-fixer.py, run it as cron job: #!/usr/bin/python target = "/Applications/Spotify.app/Contents/MacOS/Spotify" with open(target) as infile: bytes = infile.read() with open("./Spotify", "wb") as backup: backup.write(bytes) fixed = bytes.replace("VACUUM;", "xxxxxx;") with open (target, "wb") as outfile: outfile.write(fixed)
  - toomuchtodo 10y ago
    [retracted] Thanks logicallee! I definitely need to be more judicious about time management, so I appreciated your comment!
    - logicallee 10y ago
      dude. stop. the reason your username is "toomuchtodo" is shit like this. Just do it RIGHT NOW, or don't do it. It's not that important. Delete that stupid bookmark. Just because something could be fun to do if you didn't have anythign else to do and were bored out of your mind, doesn't mean it's worth doing, ever.
      - mynameisvlad 10y ago
        Is the vitrol really necessary? How does what a complete stranger decides to do in their spare time affect you in any way shape or form?
        smhost 10y ago
        I think it's more tough love than it is vitriol.
tomovo 10y ago
Hex Fiend is another hex editor for the Mac. Native and fast. http://ridiculousfish.com/hexfiend/ http://ridiculousfish.com/hexfiend/
- barbs 10y ago
  If you're looking for vi, but for binary, bvi does the job. http://bvi.sourceforge.net/ http://bvi.sourceforge.net/
  - kovrik 10y ago
    Or use hexl-mode in Emacs
  - JadeNB 10y ago
    I'm pretty sure vi itself, or at least vim, is fine with binary files. You can run it as `vim -b` or `:set binary`.
- arcticbull 10y ago
  I've always liked 0xED (http://www.suavetech.com/0xed/0xed.html http://www.suavetech.com/0xed/0xed.html)
- wamatt 10y ago
  Or do it with a one liner perl -pi -e 's/VACUUM/xxxxxx/g' /Applications/Spotify.app/Contents/MacOS/Spotify
milankragujevic 10y ago
Hmmm, could it be an issue with Chromium Embedded Framework, give that it uses SQLite and it's in libcef.dll? I uninstalled Spotify from my MBA but I'm not sure whether the damage was already done...
jhack 10y ago
Great fix. Only 1.7MB written in about 45 minutes.
deleted 10y ago
[deleted]
agopaul 10y ago
iHex is an alternative (native) hex editor and it's in the Mac App Store https://itunes.apple.com/us/app/ihex-hex-editor/id909566003?mt=12 https://itunes.apple.com/us/app/ihex-hex-editor/id909566003?...
voltagex_ 10y ago
That's a pretty cool fix. I wonder how they found that. This also highlights another issue with closed source software (and a few open source apps too): complete opaqueness. Yeah, hide the logging and status by default but at least give me a way to see what's actually going on. The Spotify Android app is worse. I wonder if it also suffers from this issue (I swapped to Apple Music after losing all of my offline tracks in Spotify again)
- 72deluxe 10y ago
  They probably just monitored what files the spotify client had open, or on Windows uses something like Process Monitor and monitored the program's activities. It would have seen a file open command pretty soon. And once they saw it open the sqlite.db, all they needed to do was run sqlite3 on the command line against the DB file and see if there was history in the file. It is possible that the Android client does the same as sqlite3 is the underlying database format for apps and is very easy to use.
heheocoenev 10y ago
How does this not break the signature on the binary and stop execution?
- toyg 10y ago
  is Spotify signed at all? Anyway, worked for me (used '------;' with the python script above).
  - brazzledazzle 10y ago
    It's signed on my laptop.
- brazzledazzle 10y ago
  Mine is signed. Maybe the HN crowd tends to have less restrictive GateKeeper settings since running unsigned stuff is kind of a pain in the ass.
joemi 10y ago
Does anyone know why "xxxxxx;"? Seems like that would make an invalid call, so wouldn't it make more sense to replace it with some sort of no-op? Or is xxxxxx sql's way of doing a no-op?
- jleahy 10y ago
  Presumably it results in an error, but if the return value isn't checked then that's fine.
- pbhjpbhj 10y ago
  DO 0 or SET @noop=0 or NULL; -- works in pg at least seem like good possibilities.
- CGamesPlay 10y ago
  Because that breaks the statement, Spotify doesn't care if this particular statement succeeds or fails, and because it has the same length as "VACUUM;". Since this is being modified in a binary, you can't change the length of the string, or else you'll overwrite some other part of the data.
- CGamesPlay 10y ago
  Because that breaks the statement, Spotify doesn't care if this particular statement succeeds or fails, and because it has the same length as "VACUUM;". Since this is being modified in a binary, you can't change the length of the string, or else you'll overwrite some other part of the data.
micaksica 10y ago
This patch also works fine on Spotify for Linux if you patch with ghex.
ulucs 10y ago
for Linux, it is at /usr/share/spotify/spotify.

ihuman 10y ago

6 or 7 x's? In your example, you used 6.

philipkglass 10y ago
7 x's, one to overwrite all of VACUUM; including the terminating semicolon. I didn't count the x's in the quoted text that I copied from the thread.
- 0x0 10y ago
  If you overwrite the ; is there not a risk you might cause the next legitimate sqlite operation to fail?
  - SturgeonsLaw 10y ago
    They might get lucky and overwrite the last SQL statement in the method. Seems likely with a statement like VACUUM.
cjg_ 10y ago
6, it wont start otherwise. I left the ; in.
- philipkglass 10y ago
  Hmm, I used 7 and my patched client is running fine. Maybe it's enough to garble VACUUM so it doesn't do anything valid.
  - cjg_ 10y ago
    If you replaced the ; also, you also didn't change the total length of the strings, I think that's what the problem is.

elsurudo 10y ago

An explanation of what this does? Also, I'm assuming this has to be done after every update, and since Spotify auto-updates, not so convenient...

philipkglass 10y ago
I didn't read all of the original thread but apparently Spotify is overusing sqlite VACUUM, causing a modest amount of data to turn into a huge write volume. https://sqlite.org/lang_vacuum.html https://sqlite.org/lang_vacuum.html
peeters 10y ago
I'm more familiar with Postgres, but it basically instructs the table storage provider to clean up unused disk. Think of it roughly as defragmentation. You're trading work (in the form of writes and CPU) for reclaimed space. Unless SQLite has some semantics around it, replacing it with "xxxxxx;" will likely just cause it to run a bad command...might be more correct to replace it with something like "------;" (comment characters).
- boulos 10y ago
  I was going to say "SELECT 1;" but it's two characters too many. Will all comment characters evaluate to whatever VACUUM does?
cosmie 10y ago
It likely just stops Spotify from VACUUM'ing the SQLite database it has. Spotify stores a local SQLite database file (by default on a Mac it's at ~/Library/Application Support/Spotify/PersistentCache/mercury.db ) The file is only about ~100MB (on my computer). With databases, you issue a vacuum command to defragment a database (reclaim space from deleted/updated rows, re-sort the data, etc). SQLite's VACUUM behavior basically just recreates the entire database from scratch in a temp file, then replaces the live file with that[1]. It makes the file smaller and more efficient to run queries against. Spotify must be triggering a VACUUM statement too aggressively (possibly after every change), rather than on a periodic schedule or after a certain amount of fragmentation. By breaking the VACUUM statement with this solution, you prevent the database from getting recreated so frequently. The file itself will grow slowly (since you're not reclaiming free space like a VACUUM would) and possibly slow down queries against the database, but that sounds like a far better tradeoff than excessive wear on the drive from recreating the file so often. [1]https://sqlite.org/lang_vacuum.html https://sqlite.org/lang_vacuum.html

Etheryte 10y ago

Use <Control-F> to search and <Control-S> to save, it seems you mixed them up in your otherwise very helpful comment. What's the source of the problem and the fix, though?

philipkglass 10y ago
Yep, mixed them up. See other comment about sqlite.

adrinavarro 10y ago

This fixes it, but it's likely any future update kills this temp fix. Hopefully Spotify will take care of this. I also noticed the extremely high writes less than a week ago. Wonder how long this has been going on.

mikeash 10y ago

Spotify updates itself fairly regularly, so you may need to reapply this fix from time to time.

philipkglass 10y ago
Brute-force solution: save this as spotify-fixer.py, run it as cron job: #!/usr/bin/python target = "/Applications/Spotify.app/Contents/MacOS/Spotify" with open(target) as infile: bytes = infile.read() with open("./Spotify", "wb") as backup: backup.write(bytes) fixed = bytes.replace("VACUUM;", "xxxxxx;") with open (target, "wb") as outfile: outfile.write(fixed)
- toomuchtodo 10y ago
  [retracted] Thanks logicallee! I definitely need to be more judicious about time management, so I appreciated your comment!
  - logicallee 10y ago
    dude. stop. the reason your username is "toomuchtodo" is shit like this. Just do it RIGHT NOW, or don't do it. It's not that important. Delete that stupid bookmark. Just because something could be fun to do if you didn't have anythign else to do and were bored out of your mind, doesn't mean it's worth doing, ever.
    - mynameisvlad 10y ago
      Is the vitrol really necessary? How does what a complete stranger decides to do in their spare time affect you in any way shape or form?
      - smhost 10y ago
        I think it's more tough love than it is vitriol.

tomovo 10y ago

Hex Fiend is another hex editor for the Mac. Native and fast. http://ridiculousfish.com/hexfiend/ http://ridiculousfish.com/hexfiend/

barbs 10y ago
If you're looking for vi, but for binary, bvi does the job. http://bvi.sourceforge.net/ http://bvi.sourceforge.net/
- kovrik 10y ago
  Or use hexl-mode in Emacs
- JadeNB 10y ago
  I'm pretty sure vi itself, or at least vim, is fine with binary files. You can run it as `vim -b` or `:set binary`.
arcticbull 10y ago
I've always liked 0xED (http://www.suavetech.com/0xed/0xed.html http://www.suavetech.com/0xed/0xed.html)
wamatt 10y ago
Or do it with a one liner perl -pi -e 's/VACUUM/xxxxxx/g' /Applications/Spotify.app/Contents/MacOS/Spotify

milankragujevic 10y ago

Hmmm, could it be an issue with Chromium Embedded Framework, give that it uses SQLite and it's in libcef.dll? I uninstalled Spotify from my MBA but I'm not sure whether the damage was already done...

jhack 10y ago

Great fix. Only 1.7MB written in about 45 minutes.

deleted 10y ago

[deleted]

agopaul 10y ago

iHex is an alternative (native) hex editor and it's in the Mac App Store https://itunes.apple.com/us/app/ihex-hex-editor/id909566003?mt=12 https://itunes.apple.com/us/app/ihex-hex-editor/id909566003?...

voltagex_ 10y ago

That's a pretty cool fix. I wonder how they found that. This also highlights another issue with closed source software (and a few open source apps too): complete opaqueness. Yeah, hide the logging and status by default but at least give me a way to see what's actually going on. The Spotify Android app is worse. I wonder if it also suffers from this issue (I swapped to Apple Music after losing all of my offline tracks in Spotify again)

72deluxe 10y ago
They probably just monitored what files the spotify client had open, or on Windows uses something like Process Monitor and monitored the program's activities. It would have seen a file open command pretty soon. And once they saw it open the sqlite.db, all they needed to do was run sqlite3 on the command line against the DB file and see if there was history in the file. It is possible that the Android client does the same as sqlite3 is the underlying database format for apps and is very easy to use.

heheocoenev 10y ago

How does this not break the signature on the binary and stop execution?

toyg 10y ago
is Spotify signed at all? Anyway, worked for me (used '------;' with the python script above).
- brazzledazzle 10y ago
  It's signed on my laptop.
brazzledazzle 10y ago
Mine is signed. Maybe the HN crowd tends to have less restrictive GateKeeper settings since running unsigned stuff is kind of a pain in the ass.

joemi 10y ago

Does anyone know why "xxxxxx;"? Seems like that would make an invalid call, so wouldn't it make more sense to replace it with some sort of no-op? Or is xxxxxx sql's way of doing a no-op?

jleahy 10y ago
Presumably it results in an error, but if the return value isn't checked then that's fine.
pbhjpbhj 10y ago
DO 0 or SET @noop=0 or NULL; -- works in pg at least seem like good possibilities.
CGamesPlay 10y ago
Because that breaks the statement, Spotify doesn't care if this particular statement succeeds or fails, and because it has the same length as "VACUUM;". Since this is being modified in a binary, you can't change the length of the string, or else you'll overwrite some other part of the data.
CGamesPlay 10y ago
Because that breaks the statement, Spotify doesn't care if this particular statement succeeds or fails, and because it has the same length as "VACUUM;". Since this is being modified in a binary, you can't change the length of the string, or else you'll overwrite some other part of the data.

micaksica 10y ago

This patch also works fine on Spotify for Linux if you patch with ghex.

ulucs 10y ago

for Linux, it is at /usr/share/spotify/spotify.