Many of these topics are constantly being championed by the somewhat obscure digital libraries space. These departments are mainly found in universities, and th

Many of these topics are constantly being championed by the somewhat obscure digital libraries space. These departments are mainly found in universities, and they focus heavily on the pipeline you described. Despite the point you made that throwing content on the internet is usually easy and "almost free", it's actually a pretty large and costly effort when you're trying to do it at scale. A lot of clerical and engineering effort goes into tagging/organizing/storing/replicating the data, guarding against bit rot and other integrity errors, and making the data easily searchable and browsable (and that's to say nothing of the political and psychological battles involved in simply getting researchers to follow through and provide the data to these processes, in the proper formats, before they finish their degree and disappear). If the hacker side of these challenges interest you, I suggest checking out the code4lib community: https://code4lib.org https://code4lib.org