4 ms·

Please breakup your main `colly.go` file into separate parts. If possible you shouldn't have a 30 line imports definition covering everything from cookies and r

Please breakup your main `colly.go` file into separate parts. If possible you shouldn't have a 30 line imports definition covering everything from cookies and regex to html and sync access. Make sure to use DNS caching on the box else add it in Go. Colly only supports a single machine via map of visited URL's. Would be great if you replace with a queue like redis or beanastalkd. visitedURLs map[uint64]bool

guessmyname 9y ago
> […] It's very helpful and healthy to have everything in a single file if you consider that manageable. SQLite has a different opinion — https://www.sqlite.org/amalgamation.html https://www.sqlite.org/amalgamation.html
- detaro 9y ago
  I don't think that's a good comparison to justify large code files: a) it's automatically generated, so you can do dev work on the "easier" split version b) Go comes with package management that helps solve the deployment issue c) Question: Does the concept of a "compilation unit" affect the Go compiler the same way?
- Xeoncross 9y ago
  > SQLite would like to disagree with you... ...Over 100 separate source files Um, how does this disagree?
fiatjaf 9y ago
Please don't follow this suggestion. It's very helpful and healthy to have everything in a single file if you consider that manageable, so no problem at all.
- JepZ 9y ago
  Well, in Go all files in one directory are part of one package (best practice) and files within the same package do not have to import each other to have access to other functions. Therefore, breaking the package into several files is common practice. A sane approach to this is for example to create a separate file for each type (Collector, HTMlElement, Request, Response, ...) and its attached functions/methods.
- gschier 9y ago
  Yes, this looks like very common Go code. No need to increase complexity by breaking it up.
- Xeoncross 9y ago
  It would be one thing if he just wanted a single go file for one reason, but the repo already contains multiple files. I'm recommending he break up the main file according to concerns. (common in go)

guessmyname 9y ago

> […] It's very helpful and healthy to have everything in a single file if you consider that manageable. SQLite has a different opinion — https://www.sqlite.org/amalgamation.html https://www.sqlite.org/amalgamation.html

detaro 9y ago
I don't think that's a good comparison to justify large code files: a) it's automatically generated, so you can do dev work on the "easier" split version b) Go comes with package management that helps solve the deployment issue c) Question: Does the concept of a "compilation unit" affect the Go compiler the same way?
Xeoncross 9y ago
> SQLite would like to disagree with you... ...Over 100 separate source files Um, how does this disagree?

fiatjaf 9y ago

Please don't follow this suggestion. It's very helpful and healthy to have everything in a single file if you consider that manageable, so no problem at all.

JepZ 9y ago
Well, in Go all files in one directory are part of one package (best practice) and files within the same package do not have to import each other to have access to other functions. Therefore, breaking the package into several files is common practice. A sane approach to this is for example to create a separate file for each type (Collector, HTMlElement, Request, Response, ...) and its attached functions/methods.
gschier 9y ago
Yes, this looks like very common Go code. No need to increase complexity by breaking it up.
Xeoncross 9y ago
It would be one thing if he just wanted a single go file for one reason, but the repo already contains multiple files. I'm recommending he break up the main file according to concerns. (common in go)