3 ms·

I should have been clearer in the question, I'm not looking for a solution to a specific problem, I'm looking for a general approach to building this kind of sy

• jey 19y ago
  Start simple: store the data in-memory in whatever way makes sense, and just write a log file that contains pickled transactions that can be played back. Conceptual sketch of how I would do it follows. This isn't "the" way to do it, just "a" way to do it. # this is the utility method used by the rest of your code # to register a new user. maybe this is at module scope in # the User.py module def register_user(user): run_transaction(AddUserTransaction(user)) # this is the actual object that represents the # transaction class AddUserTransaction(Transaction): def __init__(self, user): Transaction.__init__(self) self.user = user # all Transaction objects have an apply() method def apply(self): Transaction.apply(self) # invoke base method # the following MyGlobalUserTable probably just # stores a couple dicts as indexes over the user info # e.g. it will have a dict by username, by user ID, etc MyGlobalUserTable.add_user(self.user) def run_transaction(t): # apply the transaction first, then write to the log. # since if it crashes while running the transaction, # you don't want to crash again when you play back the log. t.apply() transaction_log.append(t) Here transaction_log.append(t) is some method that will pickle the transaction and append it to some log file. You'll have multiple classes like AddUserTransaction, all derived from Transaction. When you crash and want to play back the transaction log, all you have to do is unpickle the Transaction-derived objects and call apply() on them in the same order. http://docs.python.org/lib/module-pickle.html http://docs.python.org/lib/module-pickle.html Caveats: -- This will eventually get to have too large a startup cost to play back all the transactions, when you have a huge number of transactions. You can fix it at that point, and you can just replay all the transactions to get the data imported into the new format. -- The above approach is horrid if you want to launch a new process for every single request. You'd have to replay the transaction log for each incoming request, you'd have a data coherency nightmare, etc. So if you want to use this approach, make sure you're using something that shares one process amongst all requests. You might also need to worry about whether your server is multithreaded and in that case deal with locking and other crap. I'd suggest going simple with something like Twisted's single-request-at-a-time approach. This isn't as bad as it sounds at first; you just write everything to finish ASAP, and if there's something you need to come back to, you just request the Twisted Reactor to schedule an event. If you have some big blocking thing to do, do it in another thread then notify the Reactor when it's complete (and have the Reactor schedule an event for your code to be notified that the big blocking thing finished). As you can see, it's not totally trivial and general purpose like the RDBMS + ThingThatGeneratesNastySQLFromObjects approach (aka ORM). But you get more flexibility in your interfaces, and lower overhead. While it can be rewarding and simplifying to work in this way, it could also can lead to over-engineering and/or lost time and effort if you aren't already intimately familiar with the steps in this approach.
  • dood 19y ago
    Very instructive comment, thankyou. The need for a single process seems to be the biggest problem for me, since it prohibits hooking up this system with a framework like Pylons. Practically then, this is probably over my head right now, and overkill for my needs, but it is something I hope to explore as I learn more.
  • mechanical_fish 19y ago
    The great thing about this post is that it doesn't hide the fact that "building a site without an RDBMS" is perilously similar to "writing your own buggy, half-implemented, slow, nonstandard DBMS". Building a custom DBMS has been done, and done well, but I think a good general approach to the problem is to read the line about "over-engineering and/or lost time and effort" out loud, to your entire team, at dawn and noon and sunset on every day of the project.
    • jey 19y ago
      > is perilously similar to "writing your own buggy, half-implemented, slow, nonstandard DBMS". People have used files and in-memory data structures just fine for a long time. I don't they had bugs in their code owing solely to the fact that RDBMSes hadn't been invented yet. I also don't see how this is slow; it's all in-memory. Why bring a big honking DBMS into the picture when all you wanted was a hash table? Storing data in-memory doesn't amount to a "buggy, half-implemented, slow, nonstandard DBMS" -- it's serving an entirely different set of goals. Storing data in-memory in data structures is how programming is done. If you actually started to make the interface to your data layer as horrid as the interface to most DBMSes ($DIETY help you), you'd definitely end up with a shitty buggy half-implemented DBMS. But if you just want to store a couple hash tables with a sane interface dictated by the software design, not by your DBMS, just store the hash tables! Don't go wrestle with SQL just because that's the buzzword in-vogue. You should measure your DBMS sometime, and consider it in terms of a hash table lookup. It starts looking like a comedy of horrors: first, generate a string in an obtuse language to perform the lookup, and oh wait, don't put a quotation mark in the wrong spot! Got that string generated? Now send it over a socket to a server. Now the server is going to parse your string into an AST, turn the AST into an internal representation, then it's going to guess how to optimize your query (this is also expensive in terms of cycles). When it's done optimizing, we run it to the query evaluator which looks in memory (since your table is so tiny anyway) and pulls out your hash table value, encodes it, and ships it back over the socket to you. I agree and concede that these days you'll fit into the ecosystem better if you do use an RDBMS with no rationale (other than that it's the de facto standard), generate giant SQL strings from objects, and most worryingly from my perspective: deal with the big impedance mismatch between the two paradigms of OO/anything and RDBMS. http://en.wikipedia.org/wiki/Object-relational_impedance_mismatch http://en.wikipedia.org/wiki/Object-relational_impedance_mis... I haven't seen a single ORM implementation that really makes it truly simple. You end up manipulating your structure at a lower-level than you'd like to naturally. This is partly why my next project isn't being done with an RDBMS, and I'm not storing it just in memory because of the volume of data involved. I'm going to be using Erlang and its bundled Mnesia database. There's no impedance mismatch there. The whole thing, including the database interface, works the way Erlang works. I'll throw a party when RDBMSes die.