6 ms·
While it is fun to discuss the literal question given, this is very likely one of those "XY problems" [1]. You don't need a file with the first 300M lines cut o
by jerf 6y ago
While it is fun to discuss the literal question given, this is very likely one of those "XY problems" [1]. You don't need a file with the first 300M lines cut off. You need a view of the file with the first 300M lines cut off. To the consumer of such a file, there is no difference between that and a file with the first 300M lines cut off. One way to produce that is by producing a literal file on disk with that, but it's not the best or even IMHO a good way.
The easiest way is to use "tail -n +300000001" as someone mentioned, but then, piping the result to where it needs to go. In a bad scenario, the target program requires a file, in which case you can set up a FIFO: https://www.howtoforge.com/linux-mkfifo-command/ https://www.howtoforge.com/linux-mkfifo-command/
In the worst case, the consuming program insists on random access to the file, in which case, finally, you have the problem that you need a real (OS-level) file. But as this is a database reload situation (expand the comments in the question to see this), this is unlikely to be the case.
(In a truly desperate situation, you could grab your favorite scripting language that has FUSE support and set up a special-purpose one-file FUSE filesystem that serves that one file with the front cut off and translates all seeks correctly. It's probably easier than you think, but certainly I'd have to be backed into a corner pretty hard to do this.)
The upshot is, files aren't files, or to put it another way, the file!interface isn't the same as files!disk. File!interface is an interface that allows you to read, write, seek, etc. and files!disk are a particular implementation of a bag of bytes on disk, and despite using the same name, it's important to keep them straight. The user doesn't need a file!disk that does what need, they need a file!interface that will serve the contents they need. It is much easier to get the file!interface than the file!disk in this situation.
Or to put it another another way, it's all just numbers. Files have no ontological existence, really. It's all just a question of what numbers will be returned by a "read". If you return the right numbers, you have the "right" file, regardless of how those right numbers appeared. I realize this may sound simple and obvious when I say it, but it is a major mental block that almost all junior programmers and a fair number of mid-level programmers have. It especially causes problems when you're a network programmer doing network things; you really need to understand that in the end, it's all just numbers, and as long as you consume the correct numbers, and produce the correct numbers, the right things will happen.
(The simplest case of this was when I produced a network server on port 80 that, if you poked it with the correct magic sequence, did the thing it was supposed to do, but if you did anything else, returned a hard-coded byte string that "happened" to be an HTTP response that said what you did wrong and how to fix it, completely ignoring the rest of the "request", including ignoring if it was even HTTP at all. You don't need an HTTP library to produce an HTTP response, you just need to produce the right numbers on the TCP stream. Nginx does something similar to detect when you're trying to use HTTP on an HTTPS port, so it returns a nice message instead of a bare binary SSL negotiation.)
[1]: http://xyproblem.info/ http://xyproblem.info/ don't love the name, but it seems to be somewhat established terminology.