3 ms·

The 'big data' craze did boost graph theory funding, but the whole thing is deeply flawed at the scientific level. Most of the results are concerned with sequen

The 'big data' craze did boost graph theory funding, but the whole thing is deeply flawed at the scientific level. Most of the results are concerned with sequences of graphs that tend to infinity. It's the same thing with probability, you can handwave that 'results should be close enough' provided 'the numbers are big enough'. But it isn't easy to do so, most results in graph theory are not good enough to provide accurate lower/upper bounds. Furthermore, in probability there is the issue of determining if a theoretical distribution does match the data. With graphs, there are so many properties to look for that the question does not any answer without asserting an hypothesis such as "only assortativity and node distribution matter". One time one researcher was showing a connectivity result "showing the small-world hypothesis", which is that for any two individuals there is a chain of connections linking them of at most 5 persons who know each other on a first-name basis (formulations differ). When I asked about how the logarithmic factor in the equation, he said "it's basically the same as constant". But even more fundamentally, there are things such as 'scale economy'. The 'cost' of linking two nodes in a graph is typically more important when there are few edges than when there are many. So you shouldn't even be able to consider a typical sequence of graphs, but consider it with parameters that change value with n, which is completely out of reach of current research.

teleforce 3y ago
It's not flaw/bug it's a feature especially from engineering point of view. If you think about it LLM is a true form of big data. Not only the data is big, the memory processing requirement is big as well, and that's the very definition of big data. Because of this big data requirement, Intel latest Xeon CPU has memory fabric very close to the processor and with the advent of CXL 3.0 this new feature of high-speed memory access is going to be norm rather the exception. Another example is that big data is now trending for seismic research [1]. I'm not the author for this paper but currently we are working on offline big data for earthquakes and the results are very promising for prediction (several hours/days in advance i.e. not forecasting for months/years). Not only we use recorded big data for earthquakes in many earthquake prone countries e.g Turkey, but the technique that we are using namely time-frequency analysis requires big memory processing. Hopefully we can get fund for further research into the real-time big data collection and processing based on our offline data results. The main aim is to provide early detection and warning to the residences for the upcoming big earthquake (> 7 magnitude) with very good accuracy in a timely manner, to reduce the potential casualties. [1]Big Data Seismology: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021RG000769 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021...

teleforce 3y ago

It's not flaw/bug it's a feature especially from engineering point of view. If you think about it LLM is a true form of big data. Not only the data is big, the memory processing requirement is big as well, and that's the very definition of big data. Because of this big data requirement, Intel latest Xeon CPU has memory fabric very close to the processor and with the advent of CXL 3.0 this new feature of high-speed memory access is going to be norm rather the exception. Another example is that big data is now trending for seismic research [1]. I'm not the author for this paper but currently we are working on offline big data for earthquakes and the results are very promising for prediction (several hours/days in advance i.e. not forecasting for months/years). Not only we use recorded big data for earthquakes in many earthquake prone countries e.g Turkey, but the technique that we are using namely time-frequency analysis requires big memory processing. Hopefully we can get fund for further research into the real-time big data collection and processing based on our offline data results. The main aim is to provide early detection and warning to the residences for the upcoming big earthquake (> 7 magnitude) with very good accuracy in a timely manner, to reduce the potential casualties. [1]Big Data Seismology: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021RG000769 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021...