15 ms·
Let's Solve the File Format Problem (2019)
- jkbbwr 6y agoJust solve the php problem. `Warning: Unknown: Unable to allocate memory for pool. in Unknown on line 0`
- gre 6y agoIs that the file format problem? Are they being meta?
- ricardo81 6y agoProblem solved if they used a static page here? Guess the 'hug of death' has exhausted available memory for PHP to display the page.
- mike_d 6y agoThis is a larger ArchiveTeam problem, not a language problem. They use MediaWiki (Wikipedia) to effectively host static informational pages without a cache in front.
- thunderrabbit 6y agoFrom the cached version: > Hosting is provided gratis thanks to the kind folks at Tranquil Hosting
- mike_d 6y agoI'm not sure what point you are trying to make? If AWS gives $10,000 in cloud credits to a startup and their product goes down, it is AWS' fault?
- yashap 6y agoIndeed. I hate PHP as much as the next dev, but Varnish and the like have been around for ages, and are easy to setup and operate. Dead simple, affordable solution for an essentially static site like this.
- bawolff 6y agoTbf, the expected audience is small, its not like they are trying to sell something so its critical to stay up through some spike. The architecture is totally fine for their use case. Arcitecturing it to stand up to hn hug of death would be over-engineering and not really make sense given their goals.
- jrwr 6y agoAs the main sysadmin for the archive team's web frontend (this one happens to be hosted outside of my control) Yes, its mediawiki, no we cannot use fancy cache, JScott has a special deal with the web hoster that makes the hosting damn near bullet proof. on top of that, the overall group is anti-cloudflare. so we are kinda stuck with what Ive been able to meek out of a cpanel shared hosting plan.
- pabs3 6y agoThere are a a number of open source friendly hosters that could probably host it for free. These ones come to mind: https://osuosl.org/ https://osuosl.org/ https://www.fosshost.org/ https://www.fosshost.org/ http://www.gplhost.com/ http://www.gplhost.com/ https://www.digitalocean.com/open-source/ https://www.digitalocean.com/open-source/ https://www.gandi.net/en/gandi-supports https://www.gandi.net/en/gandi-supports
- jrwr 6y agoOh it doesn't work like that. I direct you to the time he got sued for 1 billion dollars. These are "crackpots" that love to threaten our services. He gets crazy as shit takedown notices about once a week. that he promptly ignores
- mike_d 6y agoSend me an email (in my profile), I'll give you guys a server and capacity on my CDN.
- jrwr 6y agoOh it doesn't work like that. I direct you to the time he got sued for 1 billion dollars. These are "crackpots" that love to threaten our services. He gets crazy as shit takedown notices about once a week. that he promptly ignores
- bawolff 6y agoThey could change the mediawiki config to use file cache ( https://www.mediawiki.org/wiki/Manual:$wgUseFileCache https://www.mediawiki.org/wiki/Manual:$wgUseFileCache ) which is basically poor-man's version of varnish (it saves all the rendered pages on disk, and outputs that instead of rendering the page if you're not logged in or otherwise cant have a cached page). Would probably help a lot with the hn hug of death. Not sure if it would be enough, but it might be especially if combined with different max connection settings in the webserver to prevent memory starvation.
- helloycombinat 6y agoPHP is harder than P=NP
- tinus_hn 6y agoThe solution is to not log warnings to the browser. It’s information leakage for no benefit, the users can’t fix this issue.
- ChrisMarshallNY 6y agoOh dear...
- timdorr 6y agoGoogle cache: http://webcache.googleusercontent.com/search?q=cache:G6q-II8NfeMJ:fileformats.archiveteam.org/wiki/Main_Page+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:G6q-II8... Archive.org: https://web.archive.org/web/20201128022219/http://fileformats.archiveteam.org/wiki/Main_Page https://web.archive.org/web/20201128022219/http://fileformat...
- kzrdude 6y agooh, so google cache still exists? It is not visible in search anymore, so I assumed it was dead.
- sam1r 6y agoI'm supposed to comment here and leave this link below as a clue. Two thumbs up. http://fileformats.archiveteam.org/wiki/Category:File_formats_by_extension http://fileformats.archiveteam.org/wiki/Category:File_format...
- _emacsomancer_ 6y agoSomewhat confusing is that cataloguing here has markdown and org-mode in separate sections, the former in "markup" and the later in "text-based data".
- macintux 6y agoThat makes some sense: org-mode and its ecosystem is vastly more ambitious than Markdown.
- mike_d 6y agoFrom what I can gather they are trying to compose a database of information on what application generates/can read a specific file, primarily by file extension. This already exists in libmagic (https://github.com/file/file https://github.com/file/file) and can be used on any BSD/Linux system by typing `file [filename]`
- pabs3 6y agoThe goal is a bit more expansive than libmagic, it is a wiki of file format resources, including extensions, software, format specifications, related formats and more.
- brownbat 6y agoYeah, it includes mechanical specifications of physical media so, say, alien archaeologists can get data off a VHS. People are reading the title thinking it's about filesystems. It's about cataloging all human information storage methods, from phonographs to zip drives to word perfect files.
- TAForObvReasons 6y agolibmagic focuses on identifying file type using magic numbers and other properties. The wiki has links to resources for programmatically extracting image size or color palette, for example.
- pabs3 6y agolibmagic can also extract metadata for images.
- jaclaz 6y agoAnd - in case of need - on Windows there is Trid: https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html which has also a couple online options and a database of signatures/magic numbers: https://mark0.net/soft-trid-deflist.html https://mark0.net/soft-trid-deflist.html
- tgbugs 6y agoThey say file format problem, but one of the reasons why there are so many different file formats is because current file system abstractions are fundamentally insufficient for a wide variety of use cases, particularly ones where you need to ensure that a user can't accidentally separate or delete a file. How do you keep a cache directory that backs a file with the file if the file system doesn't let you put folders "inside" of files? This is only one of many issues that have resulted in the proliferation of file formats to meet use cases with different performance characteristics that all have the common requirement for data to be bundled together an not be separable. File systems that can decouple the notion of access and data locality from inseparability simply do not exist right now, so we are left with the horrible abstraction that is the archive or zip file.
- Ericson2314 6y agoHere's to hoping https://docs.ipld.io/ https://docs.ipld.io/ helps for acylic data. Not sure what's needed for cyclic (including relational) data.
- skissane 6y ago> particularly ones where you need to ensure that a user can't accidentally separate or delete a file. How do you keep a cache directory that backs a file with the file if the file system doesn't let you put folders "inside" of files? You are worried that if an application has a directory "cache" with files in it, a user will go into that directory and randomly delete the file "cache/3F3819A1.dat", and that might confuse the application. But, suppose instead you had a single file "cache.sqlite", containing a table "cache" with an "id" column. What's to stop a user from opening that file with the sqlite command line tool and running "delete cache where id='3F3819A1'" If a user really wants to muck with your app's data by hand, you can't stop them. (Unless you go with some DRM-like solution in which your app runs in a secure enclave such as Intel SGX) Whether it is a single file or a directory of files doesn't really make a difference
- tgbugs 6y agoEven with current abstractions this is the case. I'm not worried about users that have sufficient knowledge to intentionally modify a file. Users who know enough to find, much less modify an sqlite database can make their own mess if they want to. I'm more worried about cases where something like a sidecar file gets deleted or overwritten, or fails to be copied along with the main file. Some programs and file formats come up with their own way of embedding sidecar metadata in the main file, but that is a complete hack that has to be used to work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable.
- bArray 6y agoSeems to have had the kiss of death: Warning: Unknown: Unable to allocate memory for pool. in Unknown on line 0 Warning: require(): Unable to allocate memory for pool. in /usr/local/www/mediawiki/index.php on line 54 Warning: Cannot modify header information - headers already sent in /usr/local/www/mediawiki/includes/WebStart.php on line 63 [..] Seems kinda weird this stuff isn't cached?
- floatingatoll 6y agoFwiw the content is below the errors, and is still browseable etc.
- bArray 6y agoIt depends on when more RAM becomes available. I was getting different errors and different amounts of content every refresh.
- jsz0 6y agoOpen or at least documented file formats pretty much solved the file format problem. It's incredibly rare for me to encounter any file format I can't use natively or easily convert into something else.
- jl6 6y agoFor anyone into file formats I can strongly recommend the work of Ange Albertini - polyglots, visual documentation of file layouts, and the legendary PoC||GTFO.
- greenkey 6y agoThe file format problem is that the file defines the format. Obsession with file name extensions defining format soon crumble; the .001, .002, ... being a clear case of “You know not the f- of which you speak.” To “solve” file format, you may need a lot of data beyond the file provided. Does the file come from a floppy disk labeled VisiCalc? Did you read the magnetic media correctly? Let’s say you did. You look at a file and then try, the best you can, to determine what it is, maybe given a lot of information about how the computer that read the file interpreted it. Sure, if you’re talking about more modern files, maybe the filename’s extension is a good hint. Some files contain metadata headers or footers. They maybe have multiple layers of them. That metadata may have evolved. Perhaps it wasn’t versioned. Files were stored on cassette which could be listened to as audio or interpreted as containing file definitions. What if the cassette were warped in the heat? You could still recover the file partially or fully if you knew that you could speed up portions of the recovered audio, perhaps with additional pitch-bending and error-correction. Could you do that without any background information? Maybe, but it may take much more effort.
- forgotmypw17 6y agoFor my project, I chose txt as the base format. Ascii or Unicode? There's a config for that. From that, I can go to SQLite, HTML, DAG, whatever. It's easy to work with, easy to back up and synchronize, and as human readable as it gets.
- acomjean 6y agoI like this idea of trying to solve the problem . I thought it just about disk file formats. http://fileformats.archiveteam.org/wiki/Electronic_File_Formats http://fileformats.archiveteam.org/wiki/Electronic_File_Form... At work we have one that creates a “.snapshot” directory with folders with backups. It works pretty well. (Net app?) I would like that idea at home (Apple has time machine which I guess is similar) One thing about Linux, I thought it would eventually surpass the proprietary OS in a lot of areas including file systems. I think it has in a lot of ways but file systems seem to be a bit of a mess. I’ve heard good things about BRfs and zfs but like all things Linux I’ve heard bad things (you will all your data). There are so many options and changing off the default seems hard.