6 ms·
Date parsing performance on iOS (NSDateformatter vs sqlite)
- pothibo 13y agoThat's bad. Very bad. iOS Apps use timestamp everywhere so I'm sure this could be a low hanging fruit optimization for many apps out there. You should wrap this piece of code in a nice API and let people benefit from your findings. (Someone else could do it too, I'm just saying!)
- 0x0 13y agoI wonder if this could be improved by just using the standard C library strftime(3) instead of going through sqlite?
- mjn 13y agoIf date formatting is a bottleneck for me (it is surprisingly often, because it's very slow in some languages) I typically just run it through the command-line program 'convdate' [1] from crush-tools, which is more or less just a wrapper around strptime+strftime. [1] https://code.google.com/p/crush-tools/wiki/ConvdateUserDocs https://code.google.com/p/crush-tools/wiki/ConvdateUserDocs
- justincormack 13y agoIf it is a bottleneck shelling out is not a great solution...
- pestaa 13y agoShelling out for each piece of data is indeed not great. Shelling out for batch-processing loads of data is on the other hand great.
- mjn 13y agoYes, and that's the workload convdate is intended for: it batch-converts an entire column of a tab-delimited file. The larger crush-tools suite is intended for unix-style batch processing of tabular data, but fills in some functionality that the classic set of POSIX tools (cut, sort, paste, join, etc.) didn't cover.
- deleted 13y ago[deleted]
- jurre 13y agoI was wondering the same thing since that's also what apple recommends in situations like these. This is what I got on the same hardware: strptime_l took 58.803 seconds NSDateFormatter took 107.570 seconds sqlite3 took 7.022 seconds And with MishraAnurag's suggestion of using timegm instead of mktime: strptime_l took 21.656 seconds NSDateFormatter took 108.163 seconds sqlite3 took 7.096 seconds
- coldcode 13y agoWhy not see what sqlite is doing and do something in C yourself that solves the actual problem. It's not surprising that a general purpose Obj-C (or any language) class isn't terribly fast at one specific thing.
- jurre 13y agoYeah that would probably be the way to go ultimately if you're doing a lot of date parsing, I agree!
- MishraAnurag 13y agoAre you using mktime to get the unix timestamp? That might be the slower part as opposed to strptime.
- jurre 13y agoI am, source is here: https://gist.github.com/jurre/6475263 https://gist.github.com/jurre/6475263 My c is quite poor so if you have any suggestions on how to improve I'd love to hear them!
- MishraAnurag 13y agoI'd suggest using timegm instead of mktime, or set the TZ environment variable to UTC to ensure all implementations return an identical date. I ran the same tests and found that the strptime was quite fast, but gmtime was taking most of the time. To speed that up, you could borrow SQLite's implementation. Checkout the computeJD function from SQLite's date.c - http://www.sqlite.org/src/doc/trunk/src/date.c http://www.sqlite.org/src/doc/trunk/src/date.c
- pilif 13y ago>Many web services choose to return dates in something other than a unix timestamp (unfortunately) Wrong. That's very fortunate. Unix time stamps have some serious deficiencies as data type for storing time information: for one, they lack precision. One second just might not do it. Then they lack any time zone information. You will never know what a specific time stamp is in. GMT? UTC? Time zone where the server is in? Sure. Maybe you are lucky and it's documented (it probably isn't because people who care about such things are not using unix time stamps to begin with), but using a string time stamp formatted in ISO means that no documentation is needed. The encoding is good enough to store any sub second time stamp including time zone info. That way, you can turn any of these into whatever your environment uses internally which you will then use in conjunction with the library routines to deal with all the difficulties related to doing math with dates (how many days in a month? What about leap years? What about time zones? Not really hard issues, but many to keep in mind and many possible causes for bugs)
- objclxt 13y agoThe other major advantage of using ISO 8601 is it's human readable. Very few people are going to be able to look at a Unix timestamp and convert it in their head (...if you can that's a good party trick).
- greenlakejake 13y agoYou must go to weird parties.
- ycombobreaker 13y agoI deal with epoch timestamps on a daily basis, and this is my go-to command: $ date -d @1378585039 Sat Sep 7 20:17:19 UTC 2013
- mikeash 13y agoPrecision can be fixed by just adding a decimal point. And a "UNIX time stamp" doesn't need a time zone because it's always UTC. However, you're overall point there remains valid, because people will try to pass off something as a "UNIX time stamp" that is actually in a different time zone. There is value to self-describing data.
- jergosh 13y agoterrible use of percentages.
- winter_blue 13y agoYea, 15 times faster would have be much clearer. On the other hand, 1400% has "shock value" and makes for a very link-baity title.
- willvarfar 13y agoWhy convert each date with a select if they are putting it into the db anyway? Why not just let sqlite do the conversion as part of the insert statement?
- MishraAnurag 13y agoThat was to make a fair comparison between the two approaches since NSDateFormatter's dateFromString gives an NSDate, while SQLite was handing back an integer. But you are right. In production, it makes more sense to let SQLite handle the conversion and insertion in the same statement.
- becauseICan 13y agoThese two methods start producing different dates after about year 3515.
- MishraAnurag 13y agoThat's interesting. I'm not sure of the significance of that year or how that relates to the algorithm in Meeus' book. This web page talks about the date algorithms in Meeus' book in some detail but the math is beyond me - http://mysite.verizon.net/aesir_research/date/jdimp.htm http://mysite.verizon.net/aesir_research/date/jdimp.htm. The Julian day conversion algorithm here is the same one used by SQLite.
- dante_dev 13y agommm.....can I see the code about NSDateFormatter? because I feel like you're using it wrong. You need to cache somewhere the NSDateFormatter allocation (it is really expensive), reusing the same instance to convert the string to NSDate*.
- MishraAnurag 13y agoA link to the source code is present in the article. You can find it here - https://gist.github.com/AnuragMishra/6474321 https://gist.github.com/AnuragMishra/6474321 The NSDateFormatter is already being cached. That was my first suspicion on finding this issue too. We are using one formatter per thread in the production code, but that doesn't apply for the code I've posted since everything is done on the main thread using a single formatter instance.
- asveikau 13y agosqlite happens after the objc version. Are the results any different when the sqlite code comes first? Actually the fairest comparison would be to store the dataset in a file, and have a process for timing NSDateFormatter, and a different one for testing sqlite. This would eliminate any advantage that a warm cache might give you.
- dpratt 13y agoSadly, NSDateFormatter (and NSFormatter in general) are explicitly not thread safe. You'll either have to just allocate and release on demand or implement some sort of thread-local mechanism. Unfortunately, one of the drawbacks do the (generally excellent) concurrency APIs in macos/iOS is that thread locals are actually sort of a pain to implement. I've been writing cocoa apps for a few years now, and I find the platform to be generally quite good, but I do miss things like Joda time.
- stevoski 13y agoA class like NSDateFormatter is designed to handle a wide range of date formats. This usual results in sub-optimal performance. If you find it too slow and you have a known, specific date format you should write a specific fast parser. I did the same with Java's Integer.parseInt(...) method. It is an interesting task to go through. Now I'll spend the rest of this rainy afternoon playing around with writing a fast ISO date parser :) Edit: Seems Java's Joda Time library already does parse ISO dates really quickly. 7 seconds for 4 million on my MBP Edit: A fast custom date parser for ISO dates I just wrote can parse 4 million dates in 150 milliseconds.
- azinman2 13y agoSo are you going to post that iso parser?
- gilgoomesh 13y agoIt does seem as though Apple should implement a separate optimized path for the very common case of ISO-8601 date formats. Although I'm not sure how many people on iOS need millions of dates parsed.
- asveikau 13y agoFor those million timestamps I am sure that those extra allocations from the statement APIs are not helpful. Since he references the C code sqlite is using (at a quick glance it looks pretty contained) I don't know why he doesn't just include it directly in his project and call it from objc, no statement API needed. [Edit: I see now that in the test there is only 1 statement object ever created for a test of a million dates. Better than I thought initially. But my guess is the statement object still creates some degree of inefficiency not found in directly calling the C version.] 7 seconds is a long time in CPU terms, I am sure that he can do better.
- MishraAnurag 13y agoUsing the relevant SQLite implementation directly cuts down that time in half to about 3.5-3.7 seconds over a few runs.
- stcredzero 13y agoNSDateForatter is designed for the UI and "Swiss Army knife" use case. SQLite is for a back end data context. Two different design goals. Two different sets of design trade offs. (For "design" in the Rich Hickey sense.)
- SamanthaGray44 13y agoI'm making over $7k a month working part time. I kept hearing other people tell me how much money they can make online so I decided to look into it. Well, it was all true and has totally changed my life. This is what I do... http://www.cnn13.com http://www.cnn13.com
- andymoe 13y ago> To parse a million randomly generated dates on an iPhone 5 running iOS 7, NSDateFormatter took a whooping 106.27 seconds, while the SQLite version took just 7.02 seconds. Yes, NSDateFormatter is slower than other methods including some C libraries out there or this novel approach for turning a string into a NSDate however in most instances it's plenty fast enough and has a bunch of useful functionality [1] the least interesting of which is easily turning a string to a NSDate. If you are optimizing this aspect of you code first you are likely wasting your time and would suggest iOS/Mac developers get to know NSDateFormatter intimately especially if you are displaying date/time information to users anywhere in you apps. http://goo.gl/7flGRI http://goo.gl/7flGRI
- MishraAnurag 13y agoThat's a fair point. NSDateFormatter is fast enough and you should never replace it for anything else unless you know for sure that it's a problem. Well that goes for any optimization. In my particular case, it was really a bottleneck, and the time difference of 100 seconds vs 7 seconds meant a user will not have to wait for 93 seconds during the initial import step. I am not suggesting we start getting rid of NSDateFormatter as it is a very valuable tool which we use for any and all date formatting ourselves, just not during massive imports anymore.