3 ms·
Understanding the Reaction to Amazon Prime Air
- Tepix 13y ago5kb per serialized tweet? That doesn't sound very efficient.
- csmuk 13y agoOuch. They're probably encoding in XML and several nested layers of base64!! (yes I have actually seen people doing this in a finance company's integration platform).
- waterlesscloud 13y agoTweets come with a ton of metadata these days.
- csmuk 13y agoYeah but 32x the original data size...
- jzwinck 13y agoThey're storing all the metadata, and for better or worse they are storing each string field as an "object" whereas they could have used a fixed-width field. The former of course is something like a reference count and a pointer to a string elsewhere; the latter would have some overhead when strings are of varying lengths. If memory usage were really a concern, decent savings could be realized by changing (most of?) the 13 float64 columns to float32, and some of the object (string) columns to fixed-width. For example, "lang" is two characters always, and usernames (which are stored in several fields) would be more efficiently stored as char[15]. Easy to try, just not the default behavior; I bet you could cut the total size by a third (too bad they didn't).
- csmuk 13y agoIck. I wonder if anyone has actually heard of ASN.1 BER before? Seems like ridiculous encoding and structures are the in thing at the moment.
- jzwinck 13y agoHave you used Numpy/Scipy/Pandas? It works a bit differently. It basically creates a matrix as in C--the trick is choosing the right column types. I don't see what relevance ASN.1 has here.
- csmuk 13y agoYes SciPy (although I usually end up with Mathematica).That's a generic encoding. ASN.1 is a specific and compact encoding of the data.
- mjolk 13y agoThat's a minor thing. Computers exist to do work; what matters is the result and information.
- MrUnknown 13y agoMy Reaction to Amazon Prime Air is that I live around a mile from a Amazon warehouse and I can not drive there to pick it up. Instead I need to use Free Shipping, have them "package" my purchase randomly anywhere from 2 days to 5 days so I can't cancel it on them for taking too long, then ship the item 1,000 feet or so. They could setup a location for Amazon Locker in the warehouse location for me to pickup my items. I understand I will receive absolutely no on-site customer service. They could even use the Locker for returns. I have suggested this to their support. Or, I could pay for Prime free shipping so they can mail it such a short distance and actually receive the item the next day like I do with Newegg, who is further away, which has been enjoying my business if they have the item.