fifilura

34 ms·

751.
by fifilura 2y ago
Yeah the big benefit is that it requires very little setup. You create a new partitioned table/location from the originally mapped file using a CTAS like so: CREATE TABLE new_table_name WITH ( format = 'PARQUET',
752.
by fifilura 2y ago
Pyspark is probably the way to go. I just wanted to mention that AWS Athena eats 15G parquet files for breakfast. It is trivial to map the file into Athena. But you can't connect it to anything else than file output. But it can help yo
753.
by fifilura 2y ago
Oh, that's new then? Or maybe it was a part of the monthly bill I never saw. AWS (still) costs per data loaded though.
754.
by fifilura 2y ago
What immediately makes it worth it is when the cloud provider you are (supposedly) already invested in provides the CPU cluster without extra set up cost. Examples Google BigQuery, AWS Athena or AWS serverless EMR. These cloud providers alw
755.
by fifilura 3y ago
I wonder if good REPL habits could help the ADHD brain? It still feels like you are coding so your brain is attached, but with rapid prototyping you are also designing, moving parts around to see where they would fit best.
756.
by fifilura 3y ago
I disagree! These decisions are fundamental in the engineering process. Should I use steel, concrete or wood to build this bridge? The mindless coding part starts one year later when you found that your mongoDB does not do joins, and you st
757.
by fifilura 3y ago
There is a reference to a "small grammatical error" in the rejection from Harvard, but I did not quite understand what he was referring to. Was that obvious from his reply and I missed the point? I am guessing it must be in the fo
758.
by fifilura 3y ago
That is why my comment was full of caveats! I also felt like maybe the taste was not really comparable. But I think you nailed it with the last sentence!
759.
by fifilura 3y ago
You should also read up on Nansen wintering in the Arctic. Shackleton is a story about leadership. Nansen about pure survival (and some mind boggling bravery bordering stupidity) https://en.m.wikipedia.org/wiki/Nansen%2
760.
by fifilura 3y ago
They sailed to South Georgia.
761.
by fifilura 3y ago
He was also the guy that dared to turn around and say "we'll come back another day" 180km from the south pole, a few years ahead of Amundsen and Scott. No, you must have mistaken him for Scott. Apart from the fatal final jour
762.
by fifilura 3y ago
I also made it once. The electricity for boiling all that liquid should be what makes up most of the manufacuring cost. Hm I should ask Fjällbrynt how they cope with that. If I remember correctly the trick was in the stirring. Either not st
763.
by fifilura 3y ago
I am probably way off here but I also think of Norwegian brunost or Swedish messmör. Essentially boiled down whey. It is certainly much sweeter than vege/marmite, and I am not sure the remainig taste could be classified as umami. But i
764.
by fifilura 3y ago
Yeah kafka would handle it, except in my experience i would like to avoid kafka if possible, since it adds complexity. (Fair enough it depends on how precious your data is, if it is acceptable to loose some of it if a node crashes) But some
765.
by fifilura 3y ago
Would building a data lakehouse be an option? Stream the events to s3 stored as Parquet or Avro files, maybe in Iceberg format. And then use Trino/Athena to do the long term heavy lifting. Or for on-demand use cases. Then only push wha
766.
by fifilura 3y ago
Relevant book to use as reference. Written by Simon who worked with web standards at Opera. The race to conform with whatever browser had the biggest market share that day (typically IE, or Chrome) - meant a lot of reverse engineering. htt
767.
by fifilura 3y ago
HN pro tip - scroll down past the k8s bashing to get to the real stuff. Thank you!
768.
by fifilura 3y ago
In Sweden all covers used to have holes in the upper corners. Really easy, just stick your hands in and pull the duvet out. Rumor has it thay IKEA stopped selling covers with holes at the top because lots of people came back and said they
769.
by fifilura 3y ago
I am sure this was true 2009. But I feel like the technologies that started with MapReduce have now matured into BigQuery and Presto/Trino. And those have incorporated some of the criticism in the article such as schemas. Not indexes t
770.
by fifilura 3y ago
Sometimes I think about the 16th century monarchs in France and England. About the people they were surrounded with, the rituals, and primarily about the inequality and unfairness. Map that to today and I am sure 500 years from now, people
771.
by fifilura 3y ago
Congratulations for getting the article upvoted and don't be too hard on yorself.
772.
by fifilura 3y ago
"Friends don't send parquet files to analysts who wants them in their spreadsheet program"
773.
by fifilura 3y ago
Yes. Kind of my point to. But the OP missed the possibility to step through the code.
774.
by fifilura 3y ago
Was this before BigQuery/Presto/Trino? To me it seems like those technologies would have been a good fit. They don't really work with indexes but instead regular files stored in partitions (where date is typically one of them
775.
by fifilura 3y ago
If you don't need threads in development OR production, you might as well do SELECT * from users and do the join in your imperative code. If you need threads in production I think you will end up getting rid of your for loops anyway (o
776.
by fifilura 3y ago
(apologies i did not manage to keep it). Still, sorry for that this thread derailed.
777.
by fifilura 3y ago
> you can print iteration by iteration the intermediate result You would not be able to do that with a multi-threaded/multi-process application. And this is the reason why e.g. Trino/Presto is so powerful together with SQL. Ins
778.
by fifilura 3y ago
I have one question that i could not quite find an answer to. In Flink you can set timers to wake an event up in arbitrary time without applying a window. Is this supported in Arroyo?
779.
by fifilura 3y ago
Not my proudest moment, I will punish myself with a 1 month HN hiatus. (so don't expect an answer) I guess my take was what was his reason for answering what I initially posted if all he wanted was to RTFM me?
780.
by fifilura 3y ago
I dont know how to answer this rant. Other than that you missed a chance to convince me.

More ›