4 ms·

I am a novice, and that is one of the reasons why I started this project (to learn).

danieldk 11y ago
bytes, _ := ioutil.ReadAll(os.Stdin) lines := strings.Split(string(bytes), "\n") Tip: use bufio.Scanner. scanner := bufio.NewScanner(reader) scanner.Split(bufio.ScanLines) for scanner.Scan() { // Do stuff } And you can iterate over the lines 'lazily'. If you don't want to consume \r's, make your own ScanLines :).
- deleted 11y ago
  [deleted]
  - danieldk 11y ago
    You mean bufio.ReadLine? As the documentation says, it's low-level, since you have to do the buffer allocation yourself. ReadBytes/ReadString are nicer interface-wise, but it allocates new buffers on every call. I like Scanner because it provides a nice high-level interface, but still maintains an internal buffer, reducing GC pressure. Of course, it's not one size fits all.
    - polegone 11y ago
      I tested it by on a several megabyte text file (not that large) and I can see a huge improvement in speed when I use a reader vs. loading the whole thing into memory as I did at first. I can see now how much of a difference it can make on a really large file, like in the gigabyte range. BTW, I deleted my earlier comment because the problem I had wasn't anything to do with bufio. I had just made an obvious mistake elsewhere in my code, which I've fixed now.
- Someone 11y ago
  Tail also doesn't always need to read its entire input. When given a file path, it can read backwards from the end of the file until it has found its n lines. That makes tail fast on huge files (as long as they have normal line lengths), but it also complicates the code. And you may want to mmap the file, rather than open it. Whether that is a speeds up things depends on OS, OS version, file size, file system, available memory, phase of the moon, etc.
LaFolle 11y ago
Great going. This will also help other novice Go programmers learning the language, at the same time getting a sense of how to implement their own Unix commands.
andrewchambers 11y ago
The problem is something like tail should not read the whole file into memory. tail works on a 100 gig file even with 100 megs of ram.

bytes, _ := ioutil.ReadAll(os.Stdin) lines := strings.Split(string(bytes), "\n") Tip: use bufio.Scanner. scanner := bufio.NewScanner(reader) scanner.Split(bufio.ScanLines) for scanner.Scan() { // Do stuff } And you can iterate over the lines 'lazily'. If you don't want to consume \r's, make your own ScanLines :).

deleted 11y ago
[deleted]
- danieldk 11y ago
  You mean bufio.ReadLine? As the documentation says, it's low-level, since you have to do the buffer allocation yourself. ReadBytes/ReadString are nicer interface-wise, but it allocates new buffers on every call. I like Scanner because it provides a nice high-level interface, but still maintains an internal buffer, reducing GC pressure. Of course, it's not one size fits all.
  - polegone 11y ago
    I tested it by on a several megabyte text file (not that large) and I can see a huge improvement in speed when I use a reader vs. loading the whole thing into memory as I did at first. I can see now how much of a difference it can make on a really large file, like in the gigabyte range. BTW, I deleted my earlier comment because the problem I had wasn't anything to do with bufio. I had just made an obvious mistake elsewhere in my code, which I've fixed now.
Someone 11y ago
Tail also doesn't always need to read its entire input. When given a file path, it can read backwards from the end of the file until it has found its n lines. That makes tail fast on huge files (as long as they have normal line lengths), but it also complicates the code. And you may want to mmap the file, rather than open it. Whether that is a speeds up things depends on OS, OS version, file size, file system, available memory, phase of the moon, etc.

LaFolle 11y ago

Great going. This will also help other novice Go programmers learning the language, at the same time getting a sense of how to implement their own Unix commands.

andrewchambers 11y ago

The problem is something like tail should not read the whole file into memory. tail works on a 100 gig file even with 100 megs of ram.