10 ms·
sample_contents only lists contents of 10% sample of all files. Scanning the full data set may be hard for people new to big query. I managed to query the full
by kozikow 10y ago
sample_contents only lists contents of 10% sample of all files.
Scanning the full data set may be hard for people new to big query. I managed to query the full data set in https://kozikow.wordpress.com/2016/06/29/top-emacs-packages-used-in-github-repos/ https://kozikow.wordpress.com/2016/06/29/top-emacs-packages-... . "Resources exceeded during query execution" are especially hard to debug as may mean many things that could have caused Big query to go out of memory.
Some big query tricks to make it work:
- TOP/COUNT is faster and more memory efficient than GROUP BY/ORDER
- Filtering data prior to join in sub-query reduces memory usage.
- Regexps and globs are expensive. Use LEFT/RIGHT as a faster version.
- Avoid reading all files to get around 1TB freebie scan limit. Only access file contents after filtering some paths.