afiodorov

15 ms·

121.
by afiodorov 3y ago
One sentence summary of each top level comment here: https://github.com/afiodorov/notebooks/blob/main/2023_04_30_... I wanted to read all the ideas but it became hard to read.
122.
Show HN: Save your Thoughts on Ethereum network(thoughtful3.eth.limo)
2 points by afiodorov 4y ago|0 comments
123.
by afiodorov 4y ago
You took one for the team. If the world ran on good software, you'd not cautiously deposit only $25 first and you'd not be tempted to stress-test the system with the $10k deposit. If the world ran on good software, the second $9.9
124.
by afiodorov 5y ago
made my day
125.
by afiodorov 5y ago
I have been working with data for more than 5 years now and my 2 cents is that Pandas shines with exploratory analysis. Because you have the data in memory it's easy to empirically arrive at the exact data transformation you need and f
126.
by afiodorov 6y ago
I learned C++ at the start of my software career. I am glad I did; but I am also glad I never had to write it commercially. Learning C++ makes learning other languages easier. It's C++ that made me grok the difference between stack and
127.
by afiodorov 6y ago
Location: Las Palmas, Spain Remote: Yes Willing to relocate: No Technologies: Golang, Python, GCP, Machine Learning, PostgreSQL, Microservices (Docker, Kubernetes) Résumé/CV: https://www.linkedin.com/in/tom-artiom-
128.
by afiodorov 6y ago
I have been heavily influenced by lesswrong. After the initial stage of encorporating the rationality identity and arguing about everything with everyone, I tried becoming more of an applied rationalist when I realised the first approch isn
129.
by afiodorov 6y ago
Alternatively one can load the data into Google BigQuery and start running queries without indexing the data. It's not free though, however can save a lot of time. BigQuery supports import of JSON and CSV files from Google Cloud Storag
130.
by afiodorov 6y ago
I work in data processing and I use awk occasionally to work with csv files often gigabytes in size. I join csv files that each have a header with awk '(NR == 1) || (FNR > 1)' *.csv > joined.csv Note this only works i
131.
by afiodorov 6y ago
>If IO is a bottleneck it would be much worse while trying to swap around text lines in a giant memory mapped file. There's no swapping of lines in a giant memory mapped file, the byte array of the file remains intact. The program o
132.
by afiodorov 6y ago
It's a dataset related to balance changes of bitcoin addresses downsampled to daily resolution. You could extract it from BigQuery's bitcoin public data.
133.
by afiodorov 6y ago
I had to reduce the batch-size to 512, because otherwise sort complained with batch-size too large. The timings are: real 30m54.557s user 82m9.055s sys 3m1.723s no improvement over sort without the batch-size...
134.
by afiodorov 6y ago
I agree that a significant proportion of time is spent on IO. Only 8m38s is actually spent sorting (out of 19m37s). However in the past my experiments have shown that using `sort -m` to sort chunks is much much slower than using `sort -S100
135.
by afiodorov 6y ago
this would work, but once I start sorting lines I'll need a copy of original line breaks array; so in the end I'd need 16 bytes per line anyway
136.
by afiodorov 6y ago
as I mentioned in the parent comment I tried saving just the start of each line, and then during sorting scan from the start until '\n' character, but this turned out to be much slower
137.
by afiodorov 6y ago
Could you elaborate more why this could be faster? These experiments take some time to complete. Also should I not use --parallel flag?
138.
by afiodorov 6y ago
Exact same file has been taking more than 35 minutes to sort already, so it's slower without splitting. Edit: it finished! real 35m28.370s user 40m17.129s sys 4m31.081s
139.
by afiodorov 6y ago
Indeed I load the entire file, but I think provided I have RAM to load the file with all the columns, the approach of loading everything should give me optimal performance. However I agree with your approach when there isn't enough RAM
140.
by afiodorov 6y ago
>- Saving the start and end position of a string that represents a date with 16 bytes is silly. I am not saving the start and end positions of a string that represents a data; I am saving start and end position of a line, since I need to
141.
Show HN: Radix sort big files in memory(github.com)
36 points by afiodorov 6y ago|32 comments
142.
Computing Rolling Median in Go(afiodorov.github.io)
2 points by afiodorov 7y ago|0 comments
143.
Getting a Musical Note from Audio(afiodorov.github.io)
2 points by afiodorov 7y ago|0 comments
144.
Generating Ethereum Address from Scratch(afiodorov.github.io)
78 points by afiodorov 7y ago|21 comments