12 ms·
OpenBSD's safe, new file(1) implementation
- brynet 12y agoI posted a portable version on twitter a few hours ago, but this is just something I put together quick without much testing so "YMMV". https://twitter.com/canadianbryan/status/591997995459051520 https://twitter.com/canadianbryan/status/591997995459051520
- brynet 12y agoI've updated the diff/tgz to include the new privsep functionality, systrace(4) based syscall whitelisting was conditionalized on OpenBSD.
- viraptor 12y agoWhile this simplification of tools is great, I'm a little disappointed that it's a system integrated app and it happened in OpenBSD fairly quietly. If it first happened in something like RH or Debian, the package would be separate from core and we would have the opportunity for the following: - using language with more safety guarantees than C - including seccomp (either bfp, or even just mode 1) by default - in the announcement saying: tested with XXX, YYY, ZZZ; fuzzed with ABC, DEF It would be a great opportunity to create a poster-child people could point to and say - this is the way we should be (re)writing secure software, these were the awesome tools used to help it. For OpenBSD the first two couldn't happen (part of base system so C; there's no seccomp support), the third could but didn't - at least not publicly (valgrind, clang-analyser, [am]san, afl, etc. could all be listed) Maybe the next time something important is broken.
- brynet 12y agoDon't let the work of a few OpenBSD developers discourage you from creating your own file(1) implementation using your new favourite "secure" language of the week.
- viraptor 12y agoEveryone can write their own. Not everyone can substitute the implementation used by a large distribution and make a proper announcement people will care about. I don't understand why are you mocking language choice point though. Pretty much any language which doesn't manage memory directly would be a good choice. This utility doesn't even have big performance requirements. Basically I'm saying that we can try very very hard to write secure code in languages which invite issues, or... try to eliminate whole classes or issues at a time. Is it such a terrible idea?
- brynet 12y agoNew languages and technologies come and go and you're free to use them. OpenBSD will continue to create new software while adhering to modern C coding practices, even when that means defining them.
- viraptor 12y ago> New languages and technologies You mean decades-old languages like Perl and Python for example? Technologies like the ones we can finally now afford to implement - actual syscall filtering and selective capabilities dropping? Supporting utilities which only matter at development time? How do those go away? And I wrote in the first message - OpenBSD integrates `file`, has to use C and doesn't implement seccomp. They couldn't become the poster child. But the next `file` reimplementation probably won't hit the news anymore. People can improve the way we write software right now. But choose a project at random and they don't care or don't use what's available for free. Some promotion would be great when everyone's looking.
- bandrami 12y agoYou have a lot more faith in the authors of memory management algorithms than I do.
- justincormack 12y agoWell "file" is not where I would start, as it is not really security critical for most people, although OpenBSD has good reasons. Seccomp is almost unusable for most purposes with anything resembling tight filters as you do not really know what syscalls glibc especially will add to your program. Capsicum in FreeBSD is much more usable, and they are starting to priv-sep programs with it, they are good examples, I would look at them. Allegedly Linux will get Capsicum one day. OpenBSD has many examples of priv-sep such as openssh, just using seperate processes. Linux is disadvantaged by not having a base system in teh same way - you can make a new "file" but how to know if anyone will use it. You can try to rewrite the Gnu tools better, but many are a horrible mess. You probably have to start a new Linux distro...
- TheDong 12y agoYou may find the code for the current implementation in their CVS: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/?sortby=date&only_with_tag=HEAD http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/?s... You can also find a git mirror of the referenced commit at: https://bitbucket.org/braindamaged/openbsd-src/commits/c421de15c47394de1ea2478edc0aa2c941aab2f8#chg-usr.bin/file/file.c https://bitbucket.org/braindamaged/openbsd-src/commits/c421d...
- userbinator 12y agohttp://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/file.c?rev=1.32&content-type=text/x-cvsweb-markup&sortby=date&only_with_tag=HEAD http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/file/fi... I'm not so familiar with FreeBSD coding standards and design practices but it looks like that could still be refactored even more by getting rid of all the extra memory allocations (and deallocations), making future changes less likely to introduce bugs. For example it's allocating one structure for each file given as input, and making copies of all the paths... which I'm pretty sure isn't necessary as file tests each of its input files independently; the code misleadingly implies, however, that they are somehow dependent. The old version was better in this respect. That said, compared to the much more complex old version, the rest of it is an improvement.
- rjcz 12y agoOpenBSD != FreeBSD This[0] is a good place to start. Other than than, just read the code. [0] http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man9/style.9?query=style http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man9/...
- userbinator 12y agoGetting that wrong shows how much I work with the different variants of * BSD... That's a syntactic style guide; I was referring to higher-level issues at the design level, like "avoid dynamic allocation unless absolutely necessary", which are between syntax and the general very-high-level "follow the UNIX philosophy" type of guidelines. To answer the sibling comments about preparing a patch and sending it in: this was just an observation I had from a brief glance at the code; I don't have the resources to delve into the research and find out if this is actually something they would want to change, if there is a reason behind doing it this way, or only the norm for the project (e.g. the Java community largely has a very different notion of what it means to "simplify" code) - so if someone wants to take these ideas and follow through, they are more than welcome to.
- cbd1984 12y agoWhat makes this file(1) safer than any other?
- steakejjs 12y agoLcamtuf notably found several ELF Parsing bugs in file, that appeared may have been exploitable. The work he has been doing with AFL and googles big fuzz farm, focusing on utilities that are used daily without thought is insanely important, imho
- liveoneggs 12y agolink?
- brohee 12y agoFrom afl-fuzz page (http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/): https://www.freebsd.org/security/advisories/FreeBSD-SA-14%3A16.file.asc https://www.freebsd.org/security/advisories/FreeBSD-SA-14%3A... http://bugs.gw.com/view.php?id=409 http://bugs.gw.com/view.php?id=409
- jaytaylor 12y agoThe commit message is: New implementation of the file(1) utility. This is a simplified, modernised version with a nearly complete magic(5) parser but omits some of the complex builtin tests (notably ELF) and has a reduced set of options. ok deraadt What is significance beyond the obvious? What options are now missing? What are the tradeoffs? Beyond a simplified API, are there new capabilities, functionality/use-cases, and/or improved performance?
- ploxiln 12y agobrynet, who posted below, tweeted a pdf-rendered man page for the new openbsd file command: http://brynet.biz.tm/pub/file.1.pdf http://brynet.biz.tm/pub/file.1.pdf It seems to have all the options I'd ever use. In fact I've never used any options to file (though I do use it to quickly identify a handful of files, often enough). The only reason OpenBSD does this kind of thing is because they got annoyed with the vulnerabilities in the classic file/libmagic source which everyone seems to use. They probably disliked the code style / organization of the source enough that they didn't want to patch it up (which they also do fairly often for various ports). That said I've never personally looked at the source.
- brynet 12y ago> That said I've never personally looked at the source. I'd say very few people do, which is kind of scary for a utility like file(1), which people feed any random file without giving second thought.
- elchief 12y agoSeeing as it's OpenBSD it's probably much safer than existing versions
- prodigal_erik 12y agoThere's only so much safety we should ever expect from code written in C by a human being.
- brynet 12y agoThere are many inherent problems with utilities like file(1), however, if people are going to continue using them then we should try to make the best implementation possible. This new implementation was created from scratch, carefully, with modern coding practices by a very proficient programmer.
- deleted 12y ago[deleted]
- _delirium 12y agoI do think a cleaner, more careful file(1) implementation is a step in the right direction. But it's still a bit unnerving to me that a bug in a utility like file(1), which needs to do nothing but read a file and output text to stdout, could possibly do things like overwrite other files, write to the network, etc., even in the presence of parsing bugs and a maliciously crafted file. A parsing bug in a utility that does nothing but read files and parse them should not be able to do anything worse than force a misparse! The file(1) binary simply does not need to have permission to do things like write to the network or disk. This seems like a shortcoming of the traditional Unix permissions model. A promising direction imo is the more fine-grained permission model of Solaris/Illumos, which FreeBSD's Capsicum project also aims in the direction of.
- BruceM 12y agoOr object capability systems in general ... like the material at http://erights.org/ http://erights.org/ and the security kernel work from long long ago: http://mumble.net/~jar/pubs/secureos/ http://mumble.net/~jar/pubs/secureos/ and http://mumble.net/~jar/pubs/secureos/secureos.html http://mumble.net/~jar/pubs/secureos/secureos.html
- geofft 12y agoWhile capability systems are totally awesome, for this sort of application, you don't really need it. Open the target file read-only, open the magic database read-only, keep stdout open, and then even something as restrictive as Linux's seccomp mode 1 sandbox (read(2), write(2), exit(2), nothing else) should be enough to let you print out an answer.
- thrownaway2424 12y agoCorrected headline: OpenBSD's new, untested file(1) implementation. The only thing "safe" here is that the author was completely unharmed by the unit tests that he failed to write. Where I work 2500 lines of new C code without a single line of tests would be laughed off the code review system and then probably revisited around performance review time. Heroic programming is how we got to where we are today, running the world on gigantic piles of untested gotos and question mallocs and frees. There are no real heroes in programming, only people who haven't yet figured out how to write tests.
- brohee 12y agoConsidering file is supposed to be invoked on unknown files, I fail to see how tests other than fuzzing would give any assurance security wise. A very anal code review on the other hand... And the OpenBSD people are very aware of fuzzing, this new implementation of file is a direct reaction to Michal Zalewski findings...
- randomfool 12y agoFuzzing is like rolling dice- there's only a chance it'll find the lurking issues. Unit tests enforce contracts and assist future maintenance. I sincerely hope the lack of unit tests is because it's covered by existing tests.
- e12e 12y agoWell, with tools like AFL (American Fuzzy Loop) it is more like rolling dice to find a path through a maze of numbered doors - adjusting the dice rolled (6-sided, 2-sided ...) based on how far one has come, and how many doors there is to choose from... ok, it's not really like rolling dice any more at all. Did you read: http://lcamtuf.blogspot.no/2014/11/pulling-jpegs-out-of-thin-air.html http://lcamtuf.blogspot.no/2014/11/pulling-jpegs-out-of-thin...
- Animats 12y ago"American Fuzzy Lop" - it's named after a breed of rabbit where the ears sag down.
- rhapsodyv 12y agoWhat caught my attention most was the fact they are using CVS. Nothing against, just I don't see CVS too much today. Anyone knows how much impact this change can bring? How much systems rely on file? Personally, every time I use it was as one-time script.
- bandrami 12y agoAnonCVS (the idea of a repository that J. Random user could just look at at any time) was the first innovation of OpenBSD. I guess you whipper-snappers don't remember it, but back 20 years ago for the most part unless you were on the team you would only see a project's source with each release. And several projects had an annoying tendency to take patch submissions and mutter, "yeah, um, we've fixed that already in CVS" and quietly apply your patch.
- _asummers 12y agoCVS is brought up in almost every single OpenBSD thread, right before someone complains about Comic Sans on their website. The OpenBSD guys refuse to use git because they feel it is way too complicated for the task it serves. At this point, CVS is almost their barrier to entry: if you're (royal you, not rhapsodyv) going to complain a bunch about the version control tool, you're probably not going to be an active member of their developer community. Despite using old tools, the OpenBSD guys release their updates every 6 months like clockwork, and have for the past two decades. I know of no other FOSS project with that level of project management.
- mercurial 12y ago> At this point, CVS is almost their barrier to entry: if you're (royal you, not rhapsodyv) going to complain a bunch about the version control tool, you're probably not going to be an active member of their developer community. CVS is frankly hideous. Saying "git is way too complicated" is another way of saying "CVS is way underpowered". However, a good reason to cling to this relic of a bygone age is that, as far as I know, they have everything in CVS, and it must be a lot more convenient to be able to checkout an arbitrary part of their dev tree than messing around with git submodules. That said, if their attitude is really "if you complain about CVS, you are not worthy", that is bound to turn off many people, and for good reason.