3 ms·

I had a database that was in a boot-crash loop because it had a corrupted innodb_history_list for a given table. Everything would be ok if we could just delete

I had a database that was in a boot-crash loop because it had a corrupted innodb_history_list for a given table. Everything would be ok if we could just delete the table, but that would involve opening a session and that wouldn’t be possible because of the immediate crashing. On a whim I thought, “well what if I just have a really short amount of time to connect before it reboots?” So I opened up 5 terminal windows and executed “while true; do mysql -e ‘drop table xyz;’ done” in all of them After about 10 minutes one of the hundreds of thousands of attempts to connect to this constantly rebooting database succeeded and I was able to go home at a reasonable time.

2024throwaway 3y ago
This is why remote work is so important. You could have been home the entire time.
- SCUSKU 3y ago
  To play devil's advocate, had OP not wanted to go home so badly Parkinson's Law would've kicked in and OP may have tried to do things the "right way" which may have taken much longer.
  - plugin-baby 3y ago
    Also if the DB was on-prem, then the latency when connecting from home might have been too high for the hack to work.
  - hutzlibu 3y ago
    Any ideas on what the "right way" would be in this case? To me the solution seems the most straightforward.
    - fragmede 3y ago
      Drop the table from some sort of safe mode, or figure out the bad entry in the table and hex edit the file to exclude it, or find/write some sort of recovery/fsck problem for the particular database flavor in question. Those are three alternatives that come to mind for me, which is to say, I wouldn't have thought to spam the db like that. Neat trick!
- GuB-42 3y ago
  This actually highlights a negative aspect of remote work. When you work from home, it is easy to lose track of time and ending up working the whole night. Here GP had a clear motivation, solve the problem in time to get back home and presumably disconnect. That's why I actually like to work on site on Fridays. Because I know that when I leave the office, I am done for the weekend. And if I stay for too long, security will kindly remind me that the office is closing and I should leave. So laptop turned off, in the bag, and it stays there for the weekend. Even better if Monday is also on site, since I can just leave the laptop in the office, locked away. It is a psychological trick, but it works for me. Your mileage may varry. On a more technical note, don't assume the database can be administered over the internet/VPN. Real private networks still exist.

2024throwaway 3y ago

This is why remote work is so important. You could have been home the entire time.

SCUSKU 3y ago
To play devil's advocate, had OP not wanted to go home so badly Parkinson's Law would've kicked in and OP may have tried to do things the "right way" which may have taken much longer.
- plugin-baby 3y ago
  Also if the DB was on-prem, then the latency when connecting from home might have been too high for the hack to work.
- hutzlibu 3y ago
  Any ideas on what the "right way" would be in this case? To me the solution seems the most straightforward.
  - fragmede 3y ago
    Drop the table from some sort of safe mode, or figure out the bad entry in the table and hex edit the file to exclude it, or find/write some sort of recovery/fsck problem for the particular database flavor in question. Those are three alternatives that come to mind for me, which is to say, I wouldn't have thought to spam the db like that. Neat trick!
GuB-42 3y ago
This actually highlights a negative aspect of remote work. When you work from home, it is easy to lose track of time and ending up working the whole night. Here GP had a clear motivation, solve the problem in time to get back home and presumably disconnect. That's why I actually like to work on site on Fridays. Because I know that when I leave the office, I am done for the weekend. And if I stay for too long, security will kindly remind me that the office is closing and I should leave. So laptop turned off, in the bag, and it stays there for the weekend. Even better if Monday is also on site, since I can just leave the laptop in the office, locked away. It is a psychological trick, but it works for me. Your mileage may varry. On a more technical note, don't assume the database can be administered over the internet/VPN. Real private networks still exist.