5 ms·

Business Detail #1: IT'S A LIVE ONLINE AUCTION I'm not at liberty to discuss the industries or specific applications in which this platform will be used so pl

by owkaye 18y ago

Business Detail #1: IT'S A LIVE ONLINE AUCTION I'm not at liberty to discuss the industries or specific applications in which this platform will be used so please don't ask. But I can tell you that the platform is structured like a live online auction with some unusual / unique requirements: Each auction will run for about an hour, beginning and ending at a pre-specified time. In the specific industry where we will launch this platform we can absolutely limit the number of bidders in every auction to 200,000 concurrent bidders, and we expect only about 100,000 or so bidders to be concurrently logged in and bidding.

jwilliams 18y ago
What jurisdiction will the auction be under - i.e. In most countries auctions are governed by various laws. I worked at an online, realtime auction startup in the UK in '99 (they folded) - and they encountered a whole raft of legal challenges. If so, this might influence the design. You might be in a legal predicament if (a) the system coredumps or (b) someone sues and you don't have a log of each bid or (c) both. This might be a challenge for your in-memory approach. You might need to keep a redo log that can be used to replay/verify the auction after the fact. EDIT: ps. The startup I was at used this approach. In memory, with redo logs. The client was a Java Applet - JavaScript wasn't an option in those days. Is it real money changing hands? If so you need to be concerned about security. e.g. You might need SSL or another cryptography technique. HTTPS should be fine with the JavaScript approach, but it will be a big whallop on your server requirements... You might want to consider an edge device that has SSL acceleration.
- owkaye 18y ago
  Legal and financial issues are outside the topic of this thread, but I will say this: If the auction proceeds to completion the RAM data is written to disk. This file becomes the 'auction log'. If the auction fails to complete because of technical issues we simply purge the data from RAM, re-schedule the auction and try again later. There's no need to write data to disk during the auction itself because of this 'try again' policy if anything screws up. I also think this eliminates the need for a separate re-do log since each bid can be time-stamped and stored in the auction log that's written after the auction ends -- but this brings up a question: Can timestamps be created to the nearest 1/100 of a second?
  - jwilliams 18y ago
    Fair enough - just something to be careful of. In the industry/jurisdiction I was working in, this was an issue - the view was that there was a liability if the system crashed. Perhaps in your case this doesn't apply or doesn't matter (if they are low cost items, you might just give them away for example). Anyway. In our case a bid didn't apply until the redo log had been written. So it was pseudo-transactional in that sense.
  - jwilliams 18y ago
    > Can timestamps be created to the nearest 1/100 of a second? Sure - but you need to be careful of the hardware and operating system involved. Windows tends to give lower accuracy - but there are builds of Linux that are the same.
    - owkaye 18y ago
      Thanks jwilliams, I'll more than likely be using a high performance Linux system so hopefully it will give me the precision I need.
mseebach 18y ago
I'll more-or-less repeat my advice from the other thread.. Keep the current highest bid as well as timestamp and owner of the bid in a very simple data structure in RAM. In another section of RAM, keep a datastructure for each user, keeping track of bids, and the current number of bids for the auction and the current bid. When a new bid comes in, first lookup the user's record, and check if the bid is legal, and then pass it on to compare with the current highest bid. This is your critical section. if the bid is higher than the current, update, otherwise reject.
- owkaye 18y ago
  thanks mseebach, I don't think your concept will work (and I've been known to be wrong before) ... but at least I understand how you're approaching the problem, and that helps me to get a better grasp of how I might have to go about it. It seems to me that I'll have the fastest performance by storing all the bid values in a single 'column' (in RAM of course) to make it faster to find a new unique high bid when someone posts a bid that matches the existing unique high bid. So here's my current concept of how to structure the RAM data: Create a RAM-based 'table' with only two 'columns', bidValue and bidStatus. The bidStatus column will store a single character such as 'D' for a duplicate bid, or 'U' for a unique bid, or 'H' for the unique high bid. Then go through this procedure when someone posts a new bid: - See if his bid matches the 'H' bidValue: - If so, change the bidStatus from 'H' to 'D' then identify the new unique high bid and change that bidStatus from 'U' to 'H'. - If not, and if there is already a matching bidValue in the table, set that bidStatus to 'D' - If not, and if there's no matching bidValue in the table, append a new row, then: ---------- If the bidValue in the new row is lower than the one in the 'H' row, set the bidStatus to 'U' ---------- If the bidValue in the new row is higher than the one in the 'H' row, set the bidStatus to 'H and change the old 'H' status to 'U' - Append the new bidValue, timeStamp and bidderID to another RAM-based table designed to store this data for each bidder (to create an archive of the auction). -- Send the status of the member's new bid back to him so he will see it in his browser.
  - mseebach 18y ago
    The exercise is to make your critical region (ie. the section of code that only ONE client can be executing at the same time) as small as possible, since it will be the limiting factor. If you don't think my solution will work, please explain, in terms of business requirements, why it won't - because I can't see what your alternative solution achieves, except being orders of magnitude slower then mine. For an example, you want to search for the current highest bidder. Why not always keep the current highest bidder in the same place? Looking in one place costs 1 cycle, looking in thousands of places (searching) costs thousands of cycles. Write me an email (see my profile) if you want to continue the discussion. You can NDA me if you want to.

by owkaye 18y ago

jwilliams 18y ago

What jurisdiction will the auction be under - i.e. In most countries auctions are governed by various laws. I worked at an online, realtime auction startup in the UK in '99 (they folded) - and they encountered a whole raft of legal challenges. If so, this might influence the design. You might be in a legal predicament if (a) the system coredumps or (b) someone sues and you don't have a log of each bid or (c) both. This might be a challenge for your in-memory approach. You might need to keep a redo log that can be used to replay/verify the auction after the fact. EDIT: ps. The startup I was at used this approach. In memory, with redo logs. The client was a Java Applet - JavaScript wasn't an option in those days. Is it real money changing hands? If so you need to be concerned about security. e.g. You might need SSL or another cryptography technique. HTTPS should be fine with the JavaScript approach, but it will be a big whallop on your server requirements... You might want to consider an edge device that has SSL acceleration.

owkaye 18y ago
Legal and financial issues are outside the topic of this thread, but I will say this: If the auction proceeds to completion the RAM data is written to disk. This file becomes the 'auction log'. If the auction fails to complete because of technical issues we simply purge the data from RAM, re-schedule the auction and try again later. There's no need to write data to disk during the auction itself because of this 'try again' policy if anything screws up. I also think this eliminates the need for a separate re-do log since each bid can be time-stamped and stored in the auction log that's written after the auction ends -- but this brings up a question: Can timestamps be created to the nearest 1/100 of a second?
- jwilliams 18y ago
  Fair enough - just something to be careful of. In the industry/jurisdiction I was working in, this was an issue - the view was that there was a liability if the system crashed. Perhaps in your case this doesn't apply or doesn't matter (if they are low cost items, you might just give them away for example). Anyway. In our case a bid didn't apply until the redo log had been written. So it was pseudo-transactional in that sense.
- jwilliams 18y ago
  > Can timestamps be created to the nearest 1/100 of a second? Sure - but you need to be careful of the hardware and operating system involved. Windows tends to give lower accuracy - but there are builds of Linux that are the same.
  - owkaye 18y ago
    Thanks jwilliams, I'll more than likely be using a high performance Linux system so hopefully it will give me the precision I need.

mseebach 18y ago

I'll more-or-less repeat my advice from the other thread.. Keep the current highest bid as well as timestamp and owner of the bid in a very simple data structure in RAM. In another section of RAM, keep a datastructure for each user, keeping track of bids, and the current number of bids for the auction and the current bid. When a new bid comes in, first lookup the user's record, and check if the bid is legal, and then pass it on to compare with the current highest bid. This is your critical section. if the bid is higher than the current, update, otherwise reject.

owkaye 18y ago
thanks mseebach, I don't think your concept will work (and I've been known to be wrong before) ... but at least I understand how you're approaching the problem, and that helps me to get a better grasp of how I might have to go about it. It seems to me that I'll have the fastest performance by storing all the bid values in a single 'column' (in RAM of course) to make it faster to find a new unique high bid when someone posts a bid that matches the existing unique high bid. So here's my current concept of how to structure the RAM data: Create a RAM-based 'table' with only two 'columns', bidValue and bidStatus. The bidStatus column will store a single character such as 'D' for a duplicate bid, or 'U' for a unique bid, or 'H' for the unique high bid. Then go through this procedure when someone posts a new bid: - See if his bid matches the 'H' bidValue: - If so, change the bidStatus from 'H' to 'D' then identify the new unique high bid and change that bidStatus from 'U' to 'H'. - If not, and if there is already a matching bidValue in the table, set that bidStatus to 'D' - If not, and if there's no matching bidValue in the table, append a new row, then: ---------- If the bidValue in the new row is lower than the one in the 'H' row, set the bidStatus to 'U' ---------- If the bidValue in the new row is higher than the one in the 'H' row, set the bidStatus to 'H and change the old 'H' status to 'U' - Append the new bidValue, timeStamp and bidderID to another RAM-based table designed to store this data for each bidder (to create an archive of the auction). -- Send the status of the member's new bid back to him so he will see it in his browser.
- mseebach 18y ago
  The exercise is to make your critical region (ie. the section of code that only ONE client can be executing at the same time) as small as possible, since it will be the limiting factor. If you don't think my solution will work, please explain, in terms of business requirements, why it won't - because I can't see what your alternative solution achieves, except being orders of magnitude slower then mine. For an example, you want to search for the current highest bidder. Why not always keep the current highest bidder in the same place? Looking in one place costs 1 cycle, looking in thousands of places (searching) costs thousands of cycles. Write me an email (see my profile) if you want to continue the discussion. You can NDA me if you want to.