12 ms·
DynamoDB cannot store empty strings
- idbehold 10y agoSame issue, different thread: https://github.com/aws/aws-sdk-js/issues/833 https://github.com/aws/aws-sdk-js/issues/833
- robbiet480 10y agoJust hit this problem again this morning. Quite annoying. I figured out what string was empty eventually, but i've solved this on other projects by just removing all keys with empty values to get around possible errors.
- idbehold 10y agoThe problem we have is that we're trying to store JSON provided by the user. Meaning that we need to persist empty strings provided by them. The only way I can see around this issue is to replace all empty strings with a special string token on the way in, and replace all those special string tokens on the way out.
- richardwhiuk 10y agoAppend all strings with a zero width space, and remove it on the way out.
- idbehold 10y agoIt's too bad DynamoDB can't just do this conversion for me.
- rjdavis3 10y agoWhile not ideal, you can create a new AttributeTransform that sets a place holder when storing into DynamoDB and removes it when pulling out of DynamoDB as part of your DynamoDBMapper instantiation. I did this to convert some String Sets (SS) in my database to String Lists (L). I almost did this same thing to fix the empty String issue but didn't have the time to implement it yet. http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/datamodeling/AttributeTransformer.html http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/ama...
- idbehold 10y agoUnfortunately I'm working in a node environment. The Java SDK for working with DynamoDB seems much nicer to work with. For example I believe that you can do transactions with the DynamoDB interface for Java.
- rjdavis3 10y agoSorry I saw the link was referencing the Java SDK so I thought you were using the same. The DynamoDBMapper Java SDK has been an easy to use and readable ORM for me. Adding annotations to define keys and attributes has been great. That said I'm not sure if JavaScript has an equivalent to the AttributeTransformer I mentioned.
- hehheh 10y agoIs the zero width space destined to be the next maligned value, comparable to null?
- deleted 10y ago[deleted]
- networked 10y agoWhat if the user has a string ending with a meaningful zero width space already stored? For example, the string could be checksummed somewhere. It would corrupt their data. If you want a kludge for this, it's better to generate a longish random string (e.g., a UUID) to indicate an empty value.
- eriknstr 10y agoWhen you get a string from the client you prepend a single zero width space. When you send a string back you strip the single leading space you added. The client will always have the exact same data back that they sent originally.
- networked 10y agoYou're right, of course. Sorry, I wasn't clear. I meant that a user might have stored a string with a zero width space at the end by the time you introduce this escaping mechanism. (I've already edited the comment to indicate this.) The same goes double if you append a common printable character. You'd have to rely on some additional indicator, such as the date and time the record with the string was stored, to know whether to unescape a string and also be sure nothing changed those date and time without escaping the data.
- eriknstr 10y agoOh, yeah, in that case I'd either if-case it by time stamp or I'd prepend a zero width space to all historical data as well. I would prefer doing the latter and would only do the former if there was some reason I couldn't do the latter, for example if I had too much historical data to able to process it (though I have a hard time imagining that happening for something so trivial as prepending a zero width space, unlike say converting thousands of hours of video which might actually be too time-consuming or computationally expensive). One issue that might arise with altering historical data that I can imagine would be if it was ever necessary to restore from backup and your backup was made before you later added the zero width space, and then you forget to add the zero width space again when you restore from backup a few months down the road. But with proper documentation and procedures that shouldn't happen.
- mikeash 10y agoUsing a printable character (or sequence of characters) seems like it would be a lot better. It'll ugly up your database, but at least it'll be obvious that there's a difference. Having all your string possess two forms which are visually identical but are not actually the same sounds unpleasant.
- mrob 10y agoI think it would be better to use something both printable and commonly used for placeholders, eg. an underscore, so it's obvious if you forget to remove it and unlikely to seriously confuse anybody.
- deleted 10y ago[deleted]
- MBCook 10y agoSee that's the best use case I've seen in this entire discussion. If you don't get to control if blank vs null is significant then I can see how you'd have a real issue on your hand. I'm guessing you have to have it in a form you can search it (so you can't just GZIP it or something like that)?
- idbehold 10y agoThis is all correct.
- taormina 10y agoI've known about this "problem" for ages, but has it really been 4.5 years? Wow.
- Just1689 10y agoI've bumped into this. It's a serious problem. If you think you're going to migrate an existing system to DynamoDB, then this is one of the things you need to plan for. The workarounds for this are rather unappealing, perhaps heretical.
- rjdavis3 10y agoWhat is the point of this trending? To try and get AWS to fix it? I must admit this was a gotcha for me at first but it is documented right on their standard Limits page. I'm guessing the fix is non-trivial or they probably would have just done it. http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html http://docs.aws.amazon.com/amazondynamodb/latest/developergu...
- idbehold 10y agoThis limitation should be in a giant alert at the top of all DynamoDB documentation. They are being misleading in their documentation: > A map is similar to a JSON object. There are no restrictions on the data types that can be stored in a map element, and the elements in a map do not have to be of the same type. > Maps are ideal for storing JSON documents in DynamoDB. The following example shows a map that contains a string, a number, and a nested list that contains another map.
- rjdavis3 10y agoI agree they should fix it but I think they have been transparent about the restriction. It is already listed within most of their documentation. http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html http://docs.aws.amazon.com/amazondynamodb/latest/APIReferenc... It is also listed as a Gotcha on the AWS Open Guide which trended on HN a few weeks ago. https://github.com/open-guides/og-aws#dynamodb https://github.com/open-guides/og-aws#dynamodb
- idbehold 10y agoAre you agreeing that they should fix the issue or their documentation suggesting that "Maps are ideal for storing JSON documents in DynamoDB"? Or both?
- rjdavis3 10y agoI would love for them to fix storage of empty string but I don't think that is going to happen given all the documentation they have around not allowing it. Must have been easier for them to just document the limitation. I do agree that they could do a better job in their documentation explaining the difference between JSON spec and their JSON equivalent that they store. It is not clear that the JSON you post is not stored in the same exact way. They could be more clear about the types they append to your data (L, S, SS, N) and then reference the link I put above about the restrictions for each type.
- jnordwick 10y agoNor can it save empty sets it appears.
- pinkskip 10y agoBefore they announce tons of new stuff to please the shareholders, there are much awaited enhancements on existing products which are much over due. Case in point this thread. Keep the developers happy. Just saying.
- spullara 10y agoOracle cannot store empty strings (they are treated as NULL).
- BinaryIdiot 10y agoAs far as I can tell Oracle can store empty strings they just happen to be treated exactly like NULL. That's at least marginally better. Still, it's hard to imagine people being willing to use Oracle nowadays.
- module0000 10y ago> it's hard to imagine people being willing to use Oracle nowadays. Only when it's the lesser of the super-evils you have to pick from. I'd pick Oracle over Progress(OpenEdge), but not much else :)
- spullara 10y agoThis rears its head when you want to make a string column nullable. If you do that, you can't store empty strings in it. Really broken IMHO.
- mikeash 10y agoDoes that mean that two empty strings won't compare equal to each other?
- pjmlp 10y ago> Still, it's hard to imagine people being willing to use Oracle nowadays. I use Oracle every day, because that is one of the options for our customers rely on to run their critical Fortune 500 infrastructure. The other being SQL Server.
- kpil 10y agoWhy? Because you think sql is not good or because you think there are better relational databases? I'd say that Oracle delivers quite much value for your money for many applications. I have worked with all other relevant relational databases and also with a few nosql databases, in distributed systems and monoliths. It's hard to make a recommendation that can be applied generically. The best answer is "It depends, and it's complicated" One thing is sure. Nosql is not the future (quite the opposite) although sometimes -in rather special circumstances- the right solution. Graph databases are perhaps more interesting, and perhaps event sourcing, but the promises are way larger than the deliveries. Edit. I forgot the obvious reason. You try to stay clear of the Dark Side.
- ars 10y agoThis is (was) very common in database systems. Many don't distinguish empty string from NULL. Example: http://www.ibm.com/support/knowledgecenter/SSZJPZ_11.5.0/com.ibm.swg.im.iis.conn.oracon.usage.doc/topics/suppt_null_empty_whitespace.html http://www.ibm.com/support/knowledgecenter/SSZJPZ_11.5.0/com... As a workaround, if the database supports unicode, then store a "zero width space" for a blank string.
- larrik 10y agoI used DynamoDB for an inherited project around the time I had gotten really good with MongoDB. I don't have any idea why anyone would ever use DynamoDB. It's like Mongo, without any of the good parts, that you can't run locally or any non-Amazon server. I don't see the benefit. Edit: Specifically, I was using some pretty advanced aggregation stuff. At the time I used it, DynamoDB had zero aggregation whatsoever.
- kagia 10y agoYou can run it locally for development. Just look for dynamodb-local images on the docker hub.
- idbehold 10y agoTo be fair, you can run DynamoDB locally: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
- larrik 10y agoAh, I wonder if this was true a few years ago (early 2014) when I was working with it.
- niftich 10y agoDynamoDB-Local is a service that implements DynamoDB's API, including its query language, datatypes, and grammar, but stores everything in SQLite. It's great for playing with the API or substituting a mock implementation that doesn't drain your wallet, but it's very unlikely have the backend implementation and the performance characteristics -- or even all of the exact bugs -- of "real" DynamoDB.
- bgentry 10y agolast time I used this, it had many significant differences from the real DynamoDB. Different enough that I don't really understand what purpose DynamoDB Local is actually supposed to serve.
- jonthepirate 10y ago
- coltonv 10y agoI dealt with this. It's absurd. So many smart coding conventions are completely broken by this absurd convention. Don't use Dynamo unless you for some reason have to. It's half assed and not getting many updates.
- cookiecaper 10y agoDON'T use something provided by AWS?! Why, that's heresy! ;)
- user5994461 10y agoHow about BigTable and Cassandra?
- rrdharan 10y agoGoogle Cloud Bigtable will let you store empty strings as values. I'm not quite as sure about whether you can successfully set a 0-byte value and distinguish it from an empty string ['\0'] value; the answer may depend on the client library/language/API that you use to interact with the service. [Source/Disclaimer: I work on Google Cloud Bigtable.]
- hoodoof 10y agoI tried using DynamoDB. It felt really unintuitive and like I had to bend my thought process to fit it. I misconfigured some parameters and ended up with a big bill for a database that was doing pretty much nothing. I don't see the upside of DynamoDB.
- balls187 10y ago> I don't see the upside of DynamoDB. Wide column, no-sql data store where you pay for provisioned throughput, and storage over 25GB. All maintence is taken care by Amazon, no sharding, scale, patching, or tuning required. If you need a non relational data store for use inside an AWS deployment, you would be hardpressed to find a better alternative.
- eropple 10y ago> If you need a non relational data store for use inside an AWS deployment, you would be hardpressed to find a better alternative. RDS Postgres supports HStore and JSON types.
- balls187 10y agoTo utilize it in a production environment, you would need: 1. have at minimum 2 instances 2. schedule backups 3. create and manage your db schema 4. manage users/roles Admittedly RDS makes some of those tasks easy, but Dynamodb makes it even easier. 1. Create the table, provision read/write throughput 2. Create IAM role.
- optimuspaul 10y agoWhy is this news? It might be annoying but it's super easy to work around. It makes sense to not store empty strings, they provide no value and it would only take up space.
- mikeash 10y agoHow do empty strings provide no value? It's a valid value for strings, one which many programs encounter regularly, and it's reasonable to want to store them. Refusing to store empty strings makes as much sense as refusing to store zeroes in a numeric field, because zero is "nothing."
- OJFord 10y ago> It makes sense to not store empty strings, they provide no value `""`, just like `NULL`, is semantically difference from absence of record.
- IanDrake 10y ago"As a broader question, do you have a specific use case for having an empty string in an AttributeValue?" Can anyone explain to me why this isn't a bizarre question? They can't imagine a use-case for empty strings on their own?
- deleted 10y ago[deleted]
- tyingq 10y agoI would quote them back their own marketing: "Using the AWS SDK you can write applications that store JSON documents directly into Amazon DynamoDB tables"
- PerfectElement 10y agoI recommend people look into Azure's DocumentDB. I've used Dynamo before and even though it was fast and reliable, development experience was a pain. I'm slowly migrating a self-hosted MongoDB database to DocumentDB and things are going pretty well. The only thing missing for me is aggregation and transparent encryption, but they are working on those.
- dynamothrowaway 10y agoIt also doesn't store empty lists, sets, maps, etc, not just top level, but nested inside dynamo maps. The particular SDK i was using silently removes keys with empty (whatever) values from the map. Dynamo maps in general were full of gotchas and we ended up just serializing as json since the only gotcha there is integer keys becoming strings. Also dynamo's capacity provisioning can be scaled up any time (although scaling up is slow-ish), but only down once per day, which means that unlike the promise of ec2 you end up having to essentially pay for your peak load at all times. If I had it to do over again I'd have stuck with postgres, the only reason we used dynamo in the first place was to throw our boss a bone to soften our flat refusal to use Amazon Simple Workflow, which is an even bigger quagmire.
- zwily 10y agoYou can scale down a table 4 times per day. Totally agree on SWF. :)
- MBCook 10y agoCan someone give me some examples of cases where there is a strong semantic difference that you'd want to represent with a blank string vs a null? I'm interested in abstract concepts, not "I'm trying to port from Oracle/Mongo/whatever and this is how it works". Assume your designing against some generic record store. I'm having a ton of trouble thinking of a case where that's a big problem (again, ignoring trying to save time in porting) and I was wondering if someone could provide me a few examples.
- tedmiston 10y agoThe concept of known non-existence vs unknown existence.
- techiferous 10y agoIt's the difference between "this value is unknown" and "this value is known and empty". It's similar to why JavaScript has both null and undefined. Here's an example. Let's say a user is taking a survey and the last question is: "Please share any additional thoughts if you have them." A null could mean the user did not answer this question, probably because they haven't gotten to it yet. An empty string could mean the user was asked this question and actually didn't have anything to say.
- MBCook 10y agoThat's basically the only example I could think of: tracking if someone filled out a form field. But in that case how can you tell the difference? I mean if they never clicked in the field or tabbed into it and submitted the form is that meaningfully different from they clicked into it and didn't type anything? I can think of ways to work around it (keep a boolean for 'was filled out', a number that means 'they got up to field 17', etc.) but I'm still not sure you can draw a conclusion of 'they skipped it' vs 'they didn't enter anything'.
- techiferous 10y agoYou're assuming all the questions are being presented to the user at once. Imagine a UI that presents questions to a user one-by-one. If the user bails out of the survey early, the rest of the survey answers can be null.
- KirinDave 10y agoI'm surprised this is news in 2016. Anyone who uses DynamoDB hits this immediately and suffers a bit from it. Would you like more insider advice on using DynamoDB? My startup Level Money used it as a primary data store and has kept it throughout our lifespan. We're migrating away from it now, but I wouldn't necessarily discourage people from using it. NEVER SCAN ONE TABLE AND UPDATE ROWS AS YOU ENCOUNTER THEM, OR USE NATIVE KEY ORDERING TO TRIGGER UPDATES ANYWHERE ELSE! Secretly DynamoDB is just a bunch of SQL databases with floating masters, or so we surmise. If you iterate across things in native order without randomization and at a very high speed then you will overload individual shards. You can end up at 10X write provisioning and still get rate limit responses. Randomizing the traversal of the keyspace fixes this. Which is yeah, really really bad for sufficiently large datasets. You have to get creative to randomize it sufficiently in some cases. Still, it's quite nice to have something like DynamoDB handling your scaling early on. It's a surprisingly useful design for a database and keeps you rom over-relying on relational properties which eventually don't scale. It also forces you to develop a story for cross-table transactional queries and their failures quite early in your platform's life cycle. Forcing that discipline is almost always healthy. If you're not careful though, it becomes frightfully expensive. Before we understood why we were rate limiting we panicked and ended up with >$12k/mo in DDB costs. Not really a sustainable cost for a very small company.
- chao- 10y agoCan I ask what you are migrating away to?
- pg_is_a_butt 10y agoexcel supports empty strings
- runeks 10y agoGoogle Datastore wouldn't be a bad bet.
- KirinDave 10y agoNope! We were considering RethinkDB, but then its corporate support fell apart.
- amelius 10y agoSimple workaround: prefix every string with a fixed character. Advantage: it will leave the lexicographical sort order unchanged.
- jknoepfler 10y agoWhat is exceptionally irritating about this is that there is no valid reason for not being able to represent an empty string, and the lack of support makes the service inelegant and harder to use. 1. Empty strings are straightforward to represent. 2. Not allowing them violates the principle of least surprise, which makes the service harder to use. 3. Not supporting them adds needless complexity to client applications. 4. An empty string has obvious meaning and a place in everyday applications.
- deleted 10y ago[deleted]
- ScottBurson 10y agoIt just blows my mind that someone building a database in the 21st century doesn't know that zero is not a special case. In my book, anyone who doesn't get that is a beginning programmer, not even intermediate. I know it took people a while to get this originally. In the old days, for example, Fortran 'do' loops were always executed at least once, regardless of the values of the bounds. Nowadays we know better: you check the values at the top of the loop, not (just) the bottom, to handle the zero-iteration case correctly. But now it should be burned into every developer's brain: counts can be zero, strings can be empty, lists can be empty, etc. etc. etc., and handling these cases correctly is critical. Zero is not a special case!!!
- jaysoncena 10y agoIn my previous team, we instead use the string "null" to represent NULL. Its pretty dumb but we haven't thought of any other way. What I like about DynamoDB is that you can scale the table easily. You don't need a dedicated DBA to maintain your database. For what I don't like 1. You pay for what IOPS you allocated 2. You can't share free IOPS to other tables 3. Getting a snapshot of the whole DB is impossible, DB backups are not transactional 4. Use EMR or DataPipeline for backups 5. If you reach your IOPS limit, you need to retry your writes/updates instead of delayed ack's. Others uses libraries that limits writes but its per server and doesn't account the free writes on other servers since the accounting is on the client level
- jaclaz 10y ago>In my previous team, we instead use the string "null" to represent NULL. Its pretty dumb but we haven't thought of any other way. Yep, it is a classic, BUT (just in case): https://news.ycombinator.com/item?id=12426315 https://news.ycombinator.com/item?id=12426315
- deleted 10y ago[deleted]
- idbehold 10y agoYou could use the empty-set character "∅" instead. At least it's less likely to occur.
- brongondwana 10y agoDamn, do we have to teach you kids everything? Just prefix your values with an X. http://stackoverflow.com/questions/6852612/bash-test-for-empty-string-with-x http://stackoverflow.com/questions/6852612/bash-test-for-emp...