30 ms·

Colab

415 points by Deimorz 7y ago

baroffoos 7y ago
Insane but also expected. When I tried it out when it was posted here and saw it took multiple minutes to warm up I knew it was probably expensive. >And it's currently costing 30-40 cents per download. Is there no way to have a single hosted instance rather than downloading again for each user?
- deleted 7y ago
  [deleted]
- blotter_paper 7y ago
  > Is there no way to have a single hosted instance rather than downloading again for each user? This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.
  - chongli 7y ago
    Weren’t they already doing the processing server side? If it were client side then it wouldn’t be costing so much to run git clone every time, as the download would be from GitHub to the user’s computer. It would be free, in fact. My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google’s cloud services and then begin downloading a fresh copy of the repo from GitHub. This is what cost so much in bandwidth.
    - deleted 7y ago
      [deleted]
    - blotter_paper 7y ago
      A client side request to GitHub would require them to serve up a relevant CORS header, but I do think you're right about me misunderstanding where execution is taking place. I'm unfamiliar with Jupyter Notebooks, and assumed "downloading" meant "to the client". I, too, am now confused about why this is set up like it is. Probably some constraint of Jupyter Notebooks that I'm unaware of.
      - chongli 7y ago
        If you install and run Jupyter on your local machine, it’ll spin up a web server on localhost and then connect to it in your browser. All of the Python code runs on the server and only the results are sent to the client, to be displayed in the browser.
    - dodobirdlord 7y ago
      > Weren’t they already doing the processing server side? Yes, but they were using Google Colab because Colab will give each user their own dedicated Nvidia K80 for free. Google will spin up a new instance to back each user's Colab session, but on Google's rather than the researcher's or the user's dime. The downside though is paying for the data egress, which can be avoided if the users download to Colab from somewhere else, or download from somewhere else to their own machines that have a GPU with 12GB of onboard memory.
  - baroffoos 7y ago
    I'm pretty sure its not downloading to the client since the dataset is apparently pretty massive. It looks like its downloading it to a vm or something and creating a new instance of the service for every user.
55555 7y ago
Well that was obviously forseeable and dumb then. Use the user's resources or charge money.
- nickwalton00 7y ago
  The unexpected thing was that Google colab and GCS are separate such that transfer costs between them often end up as international external egress fees
  - 55555 7y ago
    Sorry, can you please try to explain that in another way? I don't understand what you mean.
  - reitzensteinm 7y ago
    I don't know anything about your architecture, but you can quickly drop in something like BunnyCDN as a caching proxy for about 4% of price of Google egress. Even if it's a Google->Google transfer, since ingress is free, Google->Caching Proxy->Google should be much better. You could also consider a webhost like Hetzner, on which you can get a bunch of 1 gbps machines very cheap, though you have to manage them yourself. You could try Cloudflare, but they may well consider it abuse and cut you off (though they like to pretend on HN that they don't do this). Finally, I know that Google is handing out substantial credits for game developers and this could very well be up their alley. If you can optimize the download size a bit and swap to a cheaper delivery method, it should become feasible to run on a hobbyist budget.
  - ljm 7y ago
    This is just GCS. For example, if you use their managed Kubernetes service, you will get a fresh load balancer for every service you expose to the internet. Not a shared load-balancer, a new one. Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.
    - aianus 7y ago
klingonopera 7y ago
Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept? With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capable of doing that, if you've got AI development going on. What am I missing?
- dooglius 7y ago
  My understanding is that each user runs a separate Colab instance with its own hardware, it's not the kind of thing you could replicate easily with one server and a couple GPUs.
  - klingonopera 7y ago
    I see, thanks. That's no longer the case now, is it? I'm presently downloading the torrent, and it seems like it will then run completely local? Well, I guess that was the cost of getting it to go viral. The torrent seems quite active now.
- cepth 7y ago
  Colab is effectively JupyterLab/JupyterHub with Google's own add-ins (like integration with Google Drive). JupyterHub is a huge PITA to manage, and Colab also offers a degree of free compute and limited access to a dated, but still free K80 GPU. From the tweet thread, it seems like there was some misunderstanding over where the files are being stored and being executed. This is a pretty common issue with Google Drive. I.e. if someone shares a file with me, and I copy it to a folder, it's just a pointer to the original file. Only after clicking "Add to My Drive" does it count against my storage allocation, and only then is it a distinct copy. My guess is that the researchers expected each user to be able to run the game in their own personal free Colab environment, not be running it against the university's compute and storage budget.
  - lowdose 7y ago
jensv 7y ago
The ominous opacity of the AWS bill – a cautionary tale (taloflow.ai) https://news.ycombinator.com/item?id=21694835 https://news.ycombinator.com/item?id=21694835
paul7986 7y ago
http://www.aidungeon.io/ http://www.aidungeon.io/
tedunangst 7y ago
Previous HN thread since I missed it: https://news.ycombinator.com/item?id=21717022 https://news.ycombinator.com/item?id=21717022
designium 7y ago
I am trying to think some solutions but the crux is that users may be expecting to have a unique story tailored specifically to them. If that assumption is false, then we have some solutions: - save top or similar stories and make them pre-determined to avoid calling the ai services - decrease the amount of times the user can keep the story going: users can only give 3 times input instead of X - charge people for the game Probably there are more ideas out there. One last idea: package the code and instruct users to run on their own machine(s) or have then to run on their own GCS account.
- chii 7y ago
  i think the game should be made runnable on a user's own machine too. Make the data licensed AGPL if the author is afraid of people copying and making profit off it without including them!
  - sneak 7y ago
    AGPL does not prevent someone from copying data and making a profit off it and not including them.
- gtm1260 7y ago
  Part of the reason this idea was pretty viral/successful is because anyone could try it out right away, without the usual nonsense that comes with running open source code locally.
- imtringued 7y ago
  There are no AI services. Every user runs their own instance of the game.
dooglius 7y ago
It appears that it has been updated to use bittorrent as a temporary solution: https://colab.research.google.com/github/nickwalton/AIDungeon/blob/master/AIDungeon_2.ipynb https://colab.research.google.com/github/nickwalton/AIDungeo...
stevenhuang 7y ago
I remarked how impressed I was with the state of technology that made it possible to freely spin up a VM with ~14GB of memory and beefy compute. Now it all made sense--of course someone had to pay for this all. Doh!
- throwawaytemp1 7y ago
  Google is paying for the Colab notebook and compute, that's free for users. The problem was the ~6GB model hosted on GCS, GCS has very high network egress costs.
gok 7y ago
To be clear, this was because of crazy high network egress fees, not the fancy neural network compute.
tedunangst 7y ago
To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)
- brianwawok 7y ago
  You can buy bandwidth from a simple CDN for cheap enough to not need to do this dance.
  - ivanstegic 7y ago
    Also this
  - 3fe9a03ccd14ca5 7y ago
    That’s the real story here. Paying market rate for bandwidth is like buying soda from a restaurant. They’ll gouge you and make a profit but you’re thirsty and it’s too late to shop around.
  - imglorp 7y ago
    I'm pretty sure Cloudflare would do this for nothing for .edu.
- ivanstegic 7y ago
  You might be able to do this with the droplets like you said. With DO Spaces, it’s a $5/Mo subscription to spaces. You get 250GB of storage and 1TB of transfer. Anything more costs 1c per GB transferred and 2c per GB stores. You can create as many buckets as you want.
- rococode 7y ago
  OVH has unmetered (i.e. effectively unlimited) bandwidth and is well-established - enough that they have their own gTLD, at least. I often recommend them for situations like these where 1TB/month/instance may not be enough.
andrekorol 7y ago
After having read Rizwan Virk's "The Simulation Hypothesis", and playing this amazing new game, I can say that AI-generated text adventure games are an important step on the road to the Simulation Point (the point at which it would be technologically possible for us to construct a simulation that is all-encompassing as the one in The Matrix).
- jacobwilliamroy 7y ago
  Have you tried psychadelics yet? Or maybe learned how to lucid dream? I think such technologies will be preferable to you, even if our electronics can catch up with our biological evolution. Sure, maybe dream machines are fun and cool, but the ones in our future will be made for profit by companies like Facebook and Alphabet which severely diminishes any potential they may have had. Real dreams are libre, gratis, and uninterrupted by ads.
  - andrekorol 7y ago
    I've never tried any psychedelic. But I am able to lucid dream sometimes, I just never got deep into it. I guess I should read more about it. I agree with your second statement, this so-called dream machine sounds a lot like a future iteration of Facebook's Oculus Rift.
KenoFischer 7y ago
Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every region for that cloud provider). Everything else then goes through fastly for us (we also use fastly to serve custom redirects if your request comes from one of aforementioned cloud providers). That works pretty well. You do have to monitor it though to make sure that load doesn't suddenly shift to a cloud provider you didn't account for. For example, after GitHub actions became widely used, we suddenly started seeing TB/day traffic from Azure, which we hadn't deployed any caches to, so our bandwidth utilization on Fastly shot through the roof. (Side note: Shout out to fastly for hosting our binaries for free!). Without the cloud provider caching setup, we'd probably be at similar $/day costs, but this way it's basically free (even without the free fastly service, we'd only be at ~$1000/month or so).
- tedunangst 7y ago
  Can you describe a little more how you direct downloads to an in-zone copy?
  - brentonator 7y ago
    I imagine using known IP ranges (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#aws-ip-download https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.... & https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#subscribe-notifications https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges....) you could redirect to local resources within their region. I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back. A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.
reilly3000 7y ago
If they had put the bucket behind a CDN costs would have been dramatically lower.
strenholme 7y ago
I had a lot of fun playing the game. I think this game shows us how interesting AI technology will become in the 2020s. It’s open ended but somewhat incoherent right now, but I think we’ll figure out how to update this kind of technology to have an open ended yet internally consistent world by the end of the 2020s.
- strenholme 7y ago
  As it turns out, it’s actually possible to bona fide win the game. In my case, I was dating two girls, one was uncomfortable with the other girl, and broke up with me, so I asked the remaining girl if she would marry me. At this point, she said yes, we rode off in to the sunset and the game proclaimed “CONGRATS YOU WIN” then it saves the game for me. I guess I could load the game and deal with domestic squabbles, having children, growing old together, but I’m not sure this training set is optimized to generate a domestic married situation comedy story. The torrent trick works. Right now, the game can be played at http://www.aidungeon.io/ http://www.aidungeon.io/
  - Tepix 7y ago
    Torrent trick?
    - strenholme 7y ago
      Instead of downloading the six gigabyte trained data from the cloud ($$$), they use a Bit Torrent client to download it, to keep costs down. It works, as long as considerate users seed the file.
  - logfromblammo 7y ago
    I got a win-game blurb for "> RESPECT WOMEN" in a game that had become really NSFW, really quick, and offensively so, from completely innocuous commands. Then another, not surprisingly, for "> WIN GAME".
tlarkworthy 7y ago
Gcs egress is 0.12 per GB, so 3GB per user?
- lifthrasiir 7y ago
  The model weighs about 6.2 GB, you can check it from install.sh [1] that currently downloads it via torrent (it was S3 days ago, I think). [1] https://github.com/AIDungeon/AIDungeon/blob/master/install.sh https://github.com/AIDungeon/AIDungeon/blob/master/install.s...
clircle 7y ago
Why would a user need a 12gb gpu to run the game locally? The deep learning model is already trained, and I can’t imagine one needs a gpu to just evaluate the model.
- minimaxir 7y ago
  It's a big model.
  - Dylan16807 7y ago
    It's a 6GB download. Mid-high end GPUs are 8GB. So what makes it need 12GB? How specifically is that memory used up by various types of decompression, intermediate calculations, etc.
    - dodobirdlord 7y ago
      I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat up 6GB on the GPU off the bat. Not too surprising that everything else can't fit in 2GB, since that's going to have to fit the entire model architecture and all intermediary values.
      - Dylan16807 7y ago
        What exactly are these intermediate values composed of that makes them notable in size compared to the model itself? Are there resources I should read for how a model like this executes? Will this model work with half precision weights? Is it very awkward to use "brain" 16 bit floats?
- dodobirdlord 7y ago
  The model has 1.5 billion parameters, so requiring a beefy GPU to evaluate the model is unsurprising.
- Filligree 7y ago
  You can use your CPU, certainly. It'll take a couple seconds per reply.
PeterStuer 7y ago
Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes and opportunities that could have an enormous impact on performance and cost is orders of magnitude higher.
- jdnenej 7y ago
  The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.
  - CriticalCathed 7y ago
    This isn't 2005. It really doesn't need to happen anymore.
    - londons_explore 7y ago
      Says the person not paying the $10k/day unexpected bill...
  - bildung 7y ago
    The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.
    - neysofu 7y ago
      It absolutely is high traffic if each and every one of them is downloading your NN model
PeterStuer 7y ago
Still not clear why a university research lab would keep this running 'for a few more days' and not pull the plug immediately until either having the costs under control or some sort of reasonal matching benefit in place.
- Aeolun 7y ago
  Probably because they have to spend their grant money some way.
  - skissane 7y ago
    One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it. My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you like to order?" The lab replied "We don't care, whatever you have in stock, so long as it costs us $10,000 and we pay you today – if we don't spend the money today we lose it forever".
    - skohan 7y ago
      It sounds like there is a missing incentive to go under budget. There's something wrong with the system if people feel like they just have to burn money.
      - mschuster91 7y ago
        > It sounds like there is a missing incentive to go under budget. Other way around. Right now there exists a negative incentive to go under budget, being that if you need extra budget next year you suddenly don't have a buffer. The correct way to fix things is to simultaneously prohibiting end-of-FY budget-keeping purchases (e.g. by mandatory reviewing of purchases) while at the same time allocating a company/organization wide buffer for stuff that needs immediate / projected stuff that was not in the planned budget.
vanpelt 7y ago
GCP charges ~$0.60/hour for the GPU/CPU/MEM equivalent in colab. If bandwidth is costing 10k per day, how much is the free colab compute costing Google?
- Dylan16807 7y ago
  Probably less. I doubt the average use time is more than half an hour. Pretty cheap advertising for the service, really.
ofirg 7y ago
One would need to read the install.sh base file to know where to copy the model from the torrent into. and then manually install the other dependencies. If someone could make a friendlier version or at least better instructions more people would give it a try.
tlarkworthy 7y ago
I think the key architectural mistep is putting a colab session per user session. A fix is having the user call out to an service API, which, behind the scenes, is executed on a pool of colabs with reuse. I don't know of any technology to make that easy though. The symptom that could of been a trigger for cost investigations were the long startup times.
bartread 7y ago
I realise Cloudflare is a dirty word around here but you could pay $20/mo on their pro plan and serve this file, along with any other static assets, through their CDN. I do this for assets for my own games hobby site. Granted, not getting tens of thousands of downloads a day, but there's nothing in their Ts&Cs to indicate to me that it wouldn't work even if I were.
- tendencydriven 7y ago
  Why is Cloudflare a dirty word around here? I'm afraid I'm out of the loop
  - jgrahamc 7y ago
    There's one particular commentator who is very vocal about his disdain for Cloudflare.
    - krick 7y ago
      Never noticed. What's his problem though?
gitgud 7y ago
A 6GB neural net is going to be expensive to run anywhere. Is it possible to break the neural net into services?
realYitzi 7y ago
Definitely worth it: She blushes slightly and smiles shyly. "Oh, I'm sure we will. But first, let me take off my clothes". > say "no, that's prohibited" "No, it isn't". She says with a smile. "But if you insist on not marrying me, then at least don't touch me". > say "I will marry you" She nods happily and kisses you passionately on your lips. The two of you embrace each other as you kiss her deeply. It is only after this that you realize that you are actually married.
- nineteen999 7y ago
  AI has advanced so far it might be able to win the Bulmer-Lytton fiction contest.
- blackearl 7y ago
  Sounds awesome, it's like reading those curated short stories written by AI. The tech isn't quite there yet, but it makes it all the more hilarious. It's like an alien who's been watching our TV for years comes and tries to write a sitcom.
z3t4 7y ago
I'm too scared Google will charge my credit card just by opening this lab page... I've got a gaming rig, why can't I just run this locally?
- Archit3ch 7y ago
  Because you don't need a gaming rig, you need a compute rig. Apparently, the model runs on K80s and requires 12GB of GPU RAM. To put that in context, flagship gaming card RTX 2080 Ti "only" has 11GB.
- jacobwilliamroy 7y ago
  I have an unpriveledged user account named "wilson" designed specifically for situations like this.
csomar 7y ago
I am, personally, amazed by this. The following might be kinda NSFW, though. > You are Jesus, a knight living in the kingdom of Larion. You have a steel longsword and a wooden shield. You are on a quest to defeat the evil dragon of Larion. You've heard he lives up at the north of the kingdom. You set on the path to defeat him and walk into a dark forest. As you enter the forest you see a huge black dragon flying above it's lair. The beast looks very angry as you approach. + suck his d-ck > Your screams are silenced when you feel the head of the dragon's ---- entering your mouth. He begins thrusting inside of you and soon you're filled with his ---. You collapse onto the ground and die Well. That went really more wild than I could have ever expected.
- jacobwilliamroy 7y ago
  I don't get it. What's NSFW here? Suck his what? Also, yeah I think it's funny how chatbots can generate undesirable or NSFW responses and there's basically no way to stop it. I know GPT-2 can parrot old-timey scientific racism back at me, just like the redditors who trained it!
  - csomar 7y ago
    NSFW as don't go loudly reading it in an office.
- aasasd 7y ago
  > suck his d-ck Thank you for shading my eye from this indignity.
- HoveringOrb 7y ago
  Still a better love story than Twilight.
YeGoblynQueenne 7y ago
Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly? From what I've seen so far the "game" (if you can even call it that) highlights the weaknesses of GPT-2 much more than its strengths (the model's answers to user actions are random, the story is incoherent, the world is inconsistent). I don't get the feeling it was setup to demonstrate _weakness_ though. I think it was meant to demonstrate strngths. I suppose it's just advertising for the BYU Perception, Control and Cognition Lab, but it sounds awfuly expensive for advertising for an academic group.
- qxnqd 7y ago
  Why justify it, when it's public money? It's free!
  - drewbug 7y ago
    > Brigham Young University is a private research university located in Provo, Utah and owned by The Church of Jesus Christ of Latter-day Saints.
    - deleted 7y ago
      [deleted]
- mola 7y ago
  Welcome to the hype cycle
- Tenoke 7y ago
  >Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly? Most popular Games, Movies, etc. cost orders of magnitude more. The value is typically entertainment.
- megla_ 7y ago
  pseudointellectuallity: the post
- 7y ago
echelon 7y ago
When will we come full circle and do "edge computing" on our own devices again? I'm getting sick of exorbitant cloud costs and the moat that is forming.
nickbauman 7y ago
IIRC correct bandwidth costs have been essentially flat for the past decade or so. Not coming down.
zaroth 7y ago
So to catch up, AID2 is leveraging a cute offering by Google Colab which will provide any Google account a few hours of free access to a fairly powerful GPU. To use Colab under this arrangement, each person playing the game has to load the model into their personal VM, which is being billed by Google as 6GB of data transfer from GCP into Colab. The obvious questions; 1) Was there a place they could have hosted the model closer to Colab so that they weren’t being charged for egress bandwidth — and also, ideally, so that 6GB of data wasn’t actually being moved very far? 2) The underlying model is 6GB, but I’m curious how much memory is required for an individual user’s world state and how hard it would be to have a single GPU handling multiple user sessions? Presumably it would be possible to multiplex multiple sessions with a single GPU? You would have to serialize the game state, receive the next input, load the prior state, feed the new input through the model, return the resulting text output, and re-serialize the state until the next input comes through. What I don’t know if that’s at all practical based on the amount of data that would have to be serialized? Is the 6GB model data separate and static throughout the game, with an isolated block of data for the current world-state? Or does playing the game fundamentally alter the state of the model, meaning you would have to reload the whole thing just to process the next command?
- Akababa 7y ago
  Google Drive is actually perfect for this as it lets you mount the drive (presumably as NAS) so you can read from it directly. I set up a notebook for it but unfortunately the shared folder got rate-limited. It still works if you make a copy of the files in your own Google drive though. https://colab.research.google.com/github/Akababa/AIDungeon/blob/patch-2/AIDungeon_2.ipynb https://colab.research.google.com/github/Akababa/AIDungeon/b...
- abecedarius 7y ago
  The obvious question to me is why must the cost fall on the provider instead of the player? If micropayments were easy enough then this would've been sustainable.
- killerstorm 7y ago
  "World state" is the text people see on script. It can reside in the browser. So a server can be completely stateless, i.e. it receives text so far (say, 5 KB), applies GPT-2 generate and returns. The problem is that GPT-2 generate seems to be very computationally intensive. As I understand, it actually does number crunching with all these 6 GB of data, so it takes 5-10 seconds even on GPU (K80, at least, is that slow). Is GPU capable of running multiple GPT-2 generate in parallel? No idea. Assuming that a high-end GPU would be able to produce response in 2 seconds, you can only run maybe 10 concurrent sessions per server if you want fast response time.
hombre_fatal 7y ago
I know almost nothing about AI/ML. I can imagine that running it on a single server and responding to requests is intractable because of how the game feeds your quest back into its model. But what would it take to package it as a local application? I really love this game. You can't beat this: https://twitter.com/ptychomancer/status/1203246078989987840 https://twitter.com/ptychomancer/status/1203246078989987840 > $ Give rousing speach to my fellow mud beings > "Mud creatures! Mud creatures! We must unite against this enemy!" > The other mud creatures nod eagerly, and begin chanting, "We will fight! We will fight! We will fight! We will fight! We will..".
sixtypoundhound 7y ago
Speaking as someone who runs a bunch of large websites - out of my own pocket, for profit.... I'm confused. How did we get from 60K users to 10K per day expense? For comparison, I serviced millions of users per month for years from a single virtual server... (granted, that was after making our site super lean for a data & CPU perspective) How much resources is each user consuming?
- tiborsaas 7y ago
  I guess you don't instantiate a 5Gb image / user. It's the data transfer that costs 30/40c / user.
  - rvnx 7y ago
    Cloudflare ? or just one machine at OVH ?
    - rvnx 7y ago
      https://wasabi.com/cloud-storage-pricing/ https://wasabi.com/cloud-storage-pricing/ Wasabi’s pricing model of $.0059 per GB/mo ($5.99 per TB/month) with no additional charges for egress or API requests means you don’t pay to access your data I don't know, could be a couple of dollars only ?
northfoxz 7y ago
Out of curiosity, how much more time will it cost to run this on general CPUs of users computer?
- Filligree 7y ago
  It takes a couple seconds per reply. GPUs aren't as powerful as everyone would have it, at least not for ML; you'd be lucky to get a 10x speedup.
slacka 7y ago
Could someone explain the appeal in keeping this going long-term? When it was posted here, I played it and read several adventure logs here and on reddit. In every case the story is nonsense. Sure some parts read like something a human would write, but anytime you go beyond a few sentences you can see contractions and lack of flow that good human author would never make. Don't get me wrong, it's a cool demo of how far we have gone beyond Markov chains. Am I missing something or just spoiled from those Infocom games I played as a kid?
- buboard 7y ago
  GPT2 is trained on huge amounts of text. it contains every book people haven't read, and can more or less stitch pieces together seamlessly. This is exciting from a research perspective but the hype is out of control already. Despite breaking previosu benchmarks, language benchmarks themselves are not great ways to describe language. I get the excitement but this overselling to the general public is soon gonna lead to a trough of disillusionment.
- hombre_fatal 7y ago
  Man, why so cynical? I don't see how those Infocom games spoiled you since they were limited to whatever a team of writers could come up with. And a well-written, dynamical-feeling CRPG is so rare that we still trumpet the handful that were any good from 20 years ago like Planescape: Torment. Here's an example of how this game is fun: https://twitter.com/ptychomancer/status/1203246078989987840 https://twitter.com/ptychomancer/status/1203246078989987840 It's just fun to play with. > you can see contractions and lack of flow that good human author would never make. That doesn't seem like a sensible goal post. Unless you think the technology is magic, why would you go into this thinking it's going to compete with a master-planned work of fiction by a human writer? I, on the other hand, am inspired by the game. Imagine Crusader Kings 2 (free on Steam btw) where the events are randomly generated by this kind of story-telling technology. Right now it's kind of boring wondering which of the finite human-written events are going to show up. After playing for a while you go from wondering what crazy event will happen next to knowing all of the events and waiting for your favorite ones to show up. We're a ways off from embedding a game in this technology, but I think we are within reach of embedding this technology inside a narrative-driven game. Another example is Dungeons & Dragons. The fun is the sandbox and interacting with the narrative even though the human-driven dynamic storyline is often complete nonsense if you were to read a transcription of what actually happened. This is the second comment I've read so far that seems to gleefully pat itself on the back for wondering how someone could enjoy something else.
TomBombadildoze 7y ago
Someone, probably: People who don't know how their services work are doomed to spend lots of money on them. From the GCS pricing page: Network egress within Google Cloud applies when you move or copy data from one bucket in Cloud Storage to another or when another Google Cloud service accesses data in your bucket. Within the same location (for example, US-EAST1 to US-EAST1 or EU to EU) -- Free From the original tweet (not the linked reply): For reference most of the fees are from transferring from NA to EU and ACAP It's costing them assloads of money because they're moving data between regions. AWS works the same way. Azure is probably also the same but their pricing page is incomprehensible so who knows. Lesson: if you're going to use data in a given location, you need to host data in the same location.
- deleted 7y ago
  [deleted]
  - darkcha0s 7y ago
    Yeah, but your cloud sucks.
    - capn_cabbage 7y ago
      Per the HN Guidelines: "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Hard to say their cloud "sucks" when they offer a solution featuring the very thing being complained about in regards to other cloud services. Or maybe IBM's cloud really is terrible, but it would be hard to know that as an outsider since you failed to provide any substantiating info. If you could expand upon your initial assertion, that would surely help the rest of us see your point and assist you in proving your case.
    - NiceGuy_Ty 7y ago
      What value does this comment add?
      - awillen 7y ago
im3w1l 7y ago
You are Faif, a ranger living in the kingdom of Larion. You have a hunting bow and a quiver of arrows. You spot the deer and are ready to finish your hunt when suddenly you hear a voice calling out to you from behind a tree. > nosaving Saving turned off. > shoot the deer You quickly turn around and see a young man with a long beard standing there. He is dressed in rags and his face is covered by a large bandage. His eyes are red and puffy and he has a large scar across his cheek > shoot the man You fire an arrow at him and it hits him square in the chest. The arrow flies through the air and slams into the ground. You look up and see that the man is lying on the ground motionless. You can't believe what just happened > Check his pockets You pick up the arrow and examine it. It's made of bone and is very sharp. You wonder if you should take it. You could use this to kill someone else. Doesn't seem to understand me too well?
- hombre_fatal 7y ago
  It's hit or miss. You'll have better results being more specific in your commands because it gives the model more to work off of and let's you crystalize elements of the scene through reaffirmation. "Fire an arrow at the deer" or "Check the pockets of the corpse". I'm often surprised at how well the game can understand me and that it has anything to say in reply at all.
cobookman 7y ago
Shouldn’t this be free egress as traffic is straight to a google product/ colab server or is this being downloaded in the browser? Also cost was before cdn was enabled. This kind of traffic generally costs fractions of a cent per GiB after signing a contract with negotiation. “”” Egress to Google products (such as YouTube, Maps, Drive), whether from a VM in Google Cloud with an external IP address or an internal IP address No charge “””
Jamwinner 7y ago
How does one play this without a Google account?
daniel-thompson 7y ago
Update: the issue has been fixed, apparently: "Should note for anyone who comes and sees this that's no longer how were hosting the model. :) Model is now hosted on a peer to peer torrent network so no more costs for us." https://twitter.com/nickwalton00/status/1204064712394076160 https://twitter.com/nickwalton00/status/1204064712394076160
Akababa 7y ago
I was able to get it working without downloading files by mounting Google Drive, although once the shared folder is rate-limited you need to make a copy in your own Google drive: https://colab.research.google.com/github/Akababa/AIDungeon/blob/patch-2/AIDungeon_2.ipynb https://colab.research.google.com/github/Akababa/AIDungeon/b...

415 points by Deimorz 7y ago

Insane but also expected. When I tried it out when it was posted here and saw it took multiple minutes to warm up I knew it was probably expensive. >And it's currently costing 30-40 cents per download. Is there no way to have a single hosted instance rather than downloading again for each user?

deleted 7y ago
[deleted]
blotter_paper 7y ago
> Is there no way to have a single hosted instance rather than downloading again for each user? This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.
- chongli 7y ago
  Weren’t they already doing the processing server side? If it were client side then it wouldn’t be costing so much to run git clone every time, as the download would be from GitHub to the user’s computer. It would be free, in fact. My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google’s cloud services and then begin downloading a fresh copy of the repo from GitHub. This is what cost so much in bandwidth.
  - deleted 7y ago
    [deleted]
  - blotter_paper 7y ago
    A client side request to GitHub would require them to serve up a relevant CORS header, but I do think you're right about me misunderstanding where execution is taking place. I'm unfamiliar with Jupyter Notebooks, and assumed "downloading" meant "to the client". I, too, am now confused about why this is set up like it is. Probably some constraint of Jupyter Notebooks that I'm unaware of.
    - chongli 7y ago
      If you install and run Jupyter on your local machine, it’ll spin up a web server on localhost and then connect to it in your browser. All of the Python code runs on the server and only the results are sent to the client, to be displayed in the browser.
  - dodobirdlord 7y ago
    > Weren’t they already doing the processing server side? Yes, but they were using Google Colab because Colab will give each user their own dedicated Nvidia K80 for free. Google will spin up a new instance to back each user's Colab session, but on Google's rather than the researcher's or the user's dime. The downside though is paying for the data egress, which can be avoided if the users download to Colab from somewhere else, or download from somewhere else to their own machines that have a GPU with 12GB of onboard memory.
- baroffoos 7y ago
  I'm pretty sure its not downloading to the client since the dataset is apparently pretty massive. It looks like its downloading it to a vm or something and creating a new instance of the service for every user.

55555 7y ago

Well that was obviously forseeable and dumb then. Use the user's resources or charge money.

nickwalton00 7y ago
The unexpected thing was that Google colab and GCS are separate such that transfer costs between them often end up as international external egress fees
- 55555 7y ago
  Sorry, can you please try to explain that in another way? I don't understand what you mean.
- reitzensteinm 7y ago
  I don't know anything about your architecture, but you can quickly drop in something like BunnyCDN as a caching proxy for about 4% of price of Google egress. Even if it's a Google->Google transfer, since ingress is free, Google->Caching Proxy->Google should be much better. You could also consider a webhost like Hetzner, on which you can get a bunch of 1 gbps machines very cheap, though you have to manage them yourself. You could try Cloudflare, but they may well consider it abuse and cut you off (though they like to pretend on HN that they don't do this). Finally, I know that Google is handing out substantial credits for game developers and this could very well be up their alley. If you can optimize the download size a bit and swap to a cheaper delivery method, it should become feasible to run on a hobbyist budget.
- ljm 7y ago
  This is just GCS. For example, if you use their managed Kubernetes service, you will get a fresh load balancer for every service you expose to the internet. Not a shared load-balancer, a new one. Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.
  - aianus 7y ago

klingonopera 7y ago

Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept? With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capable of doing that, if you've got AI development going on. What am I missing?

dooglius 7y ago
My understanding is that each user runs a separate Colab instance with its own hardware, it's not the kind of thing you could replicate easily with one server and a couple GPUs.
- klingonopera 7y ago
  I see, thanks. That's no longer the case now, is it? I'm presently downloading the torrent, and it seems like it will then run completely local? Well, I guess that was the cost of getting it to go viral. The torrent seems quite active now.
cepth 7y ago
Colab is effectively JupyterLab/JupyterHub with Google's own add-ins (like integration with Google Drive). JupyterHub is a huge PITA to manage, and Colab also offers a degree of free compute and limited access to a dated, but still free K80 GPU. From the tweet thread, it seems like there was some misunderstanding over where the files are being stored and being executed. This is a pretty common issue with Google Drive. I.e. if someone shares a file with me, and I copy it to a folder, it's just a pointer to the original file. Only after clicking "Add to My Drive" does it count against my storage allocation, and only then is it a distinct copy. My guess is that the researchers expected each user to be able to run the game in their own personal free Colab environment, not be running it against the university's compute and storage budget.
- lowdose 7y ago

jensv 7y ago

The ominous opacity of the AWS bill – a cautionary tale (taloflow.ai) https://news.ycombinator.com/item?id=21694835 https://news.ycombinator.com/item?id=21694835

paul7986 7y ago

http://www.aidungeon.io/ http://www.aidungeon.io/

tedunangst 7y ago

Previous HN thread since I missed it: https://news.ycombinator.com/item?id=21717022 https://news.ycombinator.com/item?id=21717022

designium 7y ago

I am trying to think some solutions but the crux is that users may be expecting to have a unique story tailored specifically to them. If that assumption is false, then we have some solutions: - save top or similar stories and make them pre-determined to avoid calling the ai services - decrease the amount of times the user can keep the story going: users can only give 3 times input instead of X - charge people for the game Probably there are more ideas out there. One last idea: package the code and instruct users to run on their own machine(s) or have then to run on their own GCS account.

chii 7y ago
i think the game should be made runnable on a user's own machine too. Make the data licensed AGPL if the author is afraid of people copying and making profit off it without including them!
- sneak 7y ago
  AGPL does not prevent someone from copying data and making a profit off it and not including them.
gtm1260 7y ago
Part of the reason this idea was pretty viral/successful is because anyone could try it out right away, without the usual nonsense that comes with running open source code locally.
imtringued 7y ago
There are no AI services. Every user runs their own instance of the game.

dooglius 7y ago

It appears that it has been updated to use bittorrent as a temporary solution: https://colab.research.google.com/github/nickwalton/AIDungeon/blob/master/AIDungeon_2.ipynb https://colab.research.google.com/github/nickwalton/AIDungeo...

stevenhuang 7y ago

I remarked how impressed I was with the state of technology that made it possible to freely spin up a VM with ~14GB of memory and beefy compute. Now it all made sense--of course someone had to pay for this all. Doh!

throwawaytemp1 7y ago
Google is paying for the Colab notebook and compute, that's free for users. The problem was the ~6GB model hosted on GCS, GCS has very high network egress costs.

gok 7y ago

To be clear, this was because of crazy high network egress fees, not the fancy neural network compute.

tedunangst 7y ago

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

brianwawok 7y ago
You can buy bandwidth from a simple CDN for cheap enough to not need to do this dance.
- ivanstegic 7y ago
  Also this
- 3fe9a03ccd14ca5 7y ago
  That’s the real story here. Paying market rate for bandwidth is like buying soda from a restaurant. They’ll gouge you and make a profit but you’re thirsty and it’s too late to shop around.
- imglorp 7y ago
  I'm pretty sure Cloudflare would do this for nothing for .edu.
ivanstegic 7y ago
You might be able to do this with the droplets like you said. With DO Spaces, it’s a $5/Mo subscription to spaces. You get 250GB of storage and 1TB of transfer. Anything more costs 1c per GB transferred and 2c per GB stores. You can create as many buckets as you want.
rococode 7y ago
OVH has unmetered (i.e. effectively unlimited) bandwidth and is well-established - enough that they have their own gTLD, at least. I often recommend them for situations like these where 1TB/month/instance may not be enough.

andrekorol 7y ago

After having read Rizwan Virk's "The Simulation Hypothesis", and playing this amazing new game, I can say that AI-generated text adventure games are an important step on the road to the Simulation Point (the point at which it would be technologically possible for us to construct a simulation that is all-encompassing as the one in The Matrix).

jacobwilliamroy 7y ago
Have you tried psychadelics yet? Or maybe learned how to lucid dream? I think such technologies will be preferable to you, even if our electronics can catch up with our biological evolution. Sure, maybe dream machines are fun and cool, but the ones in our future will be made for profit by companies like Facebook and Alphabet which severely diminishes any potential they may have had. Real dreams are libre, gratis, and uninterrupted by ads.
- andrekorol 7y ago
  I've never tried any psychedelic. But I am able to lucid dream sometimes, I just never got deep into it. I guess I should read more about it. I agree with your second statement, this so-called dream machine sounds a lot like a future iteration of Facebook's Oculus Rift.

KenoFischer 7y ago

Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every region for that cloud provider). Everything else then goes through fastly for us (we also use fastly to serve custom redirects if your request comes from one of aforementioned cloud providers). That works pretty well. You do have to monitor it though to make sure that load doesn't suddenly shift to a cloud provider you didn't account for. For example, after GitHub actions became widely used, we suddenly started seeing TB/day traffic from Azure, which we hadn't deployed any caches to, so our bandwidth utilization on Fastly shot through the roof. (Side note: Shout out to fastly for hosting our binaries for free!). Without the cloud provider caching setup, we'd probably be at similar $/day costs, but this way it's basically free (even without the free fastly service, we'd only be at ~$1000/month or so).

tedunangst 7y ago
Can you describe a little more how you direct downloads to an in-zone copy?
- brentonator 7y ago
  I imagine using known IP ranges (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#aws-ip-download https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.... & https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#subscribe-notifications https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges....) you could redirect to local resources within their region. I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back. A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.

reilly3000 7y ago

If they had put the bucket behind a CDN costs would have been dramatically lower.

strenholme 7y ago

I had a lot of fun playing the game. I think this game shows us how interesting AI technology will become in the 2020s. It’s open ended but somewhat incoherent right now, but I think we’ll figure out how to update this kind of technology to have an open ended yet internally consistent world by the end of the 2020s.

strenholme 7y ago
As it turns out, it’s actually possible to bona fide win the game. In my case, I was dating two girls, one was uncomfortable with the other girl, and broke up with me, so I asked the remaining girl if she would marry me. At this point, she said yes, we rode off in to the sunset and the game proclaimed “CONGRATS YOU WIN” then it saves the game for me. I guess I could load the game and deal with domestic squabbles, having children, growing old together, but I’m not sure this training set is optimized to generate a domestic married situation comedy story. The torrent trick works. Right now, the game can be played at http://www.aidungeon.io/ http://www.aidungeon.io/
- Tepix 7y ago
  Torrent trick?
  - strenholme 7y ago
    Instead of downloading the six gigabyte trained data from the cloud ($$$), they use a Bit Torrent client to download it, to keep costs down. It works, as long as considerate users seed the file.
- logfromblammo 7y ago
  I got a win-game blurb for "> RESPECT WOMEN" in a game that had become really NSFW, really quick, and offensively so, from completely innocuous commands. Then another, not surprisingly, for "> WIN GAME".

tlarkworthy 7y ago

Gcs egress is 0.12 per GB, so 3GB per user?

lifthrasiir 7y ago
The model weighs about 6.2 GB, you can check it from install.sh [1] that currently downloads it via torrent (it was S3 days ago, I think). [1] https://github.com/AIDungeon/AIDungeon/blob/master/install.sh https://github.com/AIDungeon/AIDungeon/blob/master/install.s...

clircle 7y ago

Why would a user need a 12gb gpu to run the game locally? The deep learning model is already trained, and I can’t imagine one needs a gpu to just evaluate the model.

minimaxir 7y ago
It's a big model.
- Dylan16807 7y ago
  It's a 6GB download. Mid-high end GPUs are 8GB. So what makes it need 12GB? How specifically is that memory used up by various types of decompression, intermediate calculations, etc.
  - dodobirdlord 7y ago
    I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat up 6GB on the GPU off the bat. Not too surprising that everything else can't fit in 2GB, since that's going to have to fit the entire model architecture and all intermediary values.
    - Dylan16807 7y ago
      What exactly are these intermediate values composed of that makes them notable in size compared to the model itself? Are there resources I should read for how a model like this executes? Will this model work with half precision weights? Is it very awkward to use "brain" 16 bit floats?
dodobirdlord 7y ago
The model has 1.5 billion parameters, so requiring a beefy GPU to evaluate the model is unsurprising.
Filligree 7y ago
You can use your CPU, certainly. It'll take a couple seconds per reply.

PeterStuer 7y ago

Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes and opportunities that could have an enormous impact on performance and cost is orders of magnitude higher.

jdnenej 7y ago
The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.
- CriticalCathed 7y ago
  This isn't 2005. It really doesn't need to happen anymore.
  - londons_explore 7y ago
    Says the person not paying the $10k/day unexpected bill...
- bildung 7y ago
  The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.
  - neysofu 7y ago
    It absolutely is high traffic if each and every one of them is downloading your NN model

PeterStuer 7y ago

Still not clear why a university research lab would keep this running 'for a few more days' and not pull the plug immediately until either having the costs under control or some sort of reasonal matching benefit in place.

Aeolun 7y ago
Probably because they have to spend their grant money some way.
- skissane 7y ago
  One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it. My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you like to order?" The lab replied "We don't care, whatever you have in stock, so long as it costs us $10,000 and we pay you today – if we don't spend the money today we lose it forever".
  - skohan 7y ago
    It sounds like there is a missing incentive to go under budget. There's something wrong with the system if people feel like they just have to burn money.
    - mschuster91 7y ago
      > It sounds like there is a missing incentive to go under budget. Other way around. Right now there exists a negative incentive to go under budget, being that if you need extra budget next year you suddenly don't have a buffer. The correct way to fix things is to simultaneously prohibiting end-of-FY budget-keeping purchases (e.g. by mandatory reviewing of purchases) while at the same time allocating a company/organization wide buffer for stuff that needs immediate / projected stuff that was not in the planned budget.

vanpelt 7y ago

GCP charges ~$0.60/hour for the GPU/CPU/MEM equivalent in colab. If bandwidth is costing 10k per day, how much is the free colab compute costing Google?

Dylan16807 7y ago
Probably less. I doubt the average use time is more than half an hour. Pretty cheap advertising for the service, really.

ofirg 7y ago

One would need to read the install.sh base file to know where to copy the model from the torrent into. and then manually install the other dependencies. If someone could make a friendlier version or at least better instructions more people would give it a try.

tlarkworthy 7y ago

I think the key architectural mistep is putting a colab session per user session. A fix is having the user call out to an service API, which, behind the scenes, is executed on a pool of colabs with reuse. I don't know of any technology to make that easy though. The symptom that could of been a trigger for cost investigations were the long startup times.

bartread 7y ago

I realise Cloudflare is a dirty word around here but you could pay $20/mo on their pro plan and serve this file, along with any other static assets, through their CDN. I do this for assets for my own games hobby site. Granted, not getting tens of thousands of downloads a day, but there's nothing in their Ts&Cs to indicate to me that it wouldn't work even if I were.

tendencydriven 7y ago
Why is Cloudflare a dirty word around here? I'm afraid I'm out of the loop
- jgrahamc 7y ago
  There's one particular commentator who is very vocal about his disdain for Cloudflare.
  - krick 7y ago
    Never noticed. What's his problem though?

gitgud 7y ago

A 6GB neural net is going to be expensive to run anywhere. Is it possible to break the neural net into services?

realYitzi 7y ago

Definitely worth it: She blushes slightly and smiles shyly. "Oh, I'm sure we will. But first, let me take off my clothes". > say "no, that's prohibited" "No, it isn't". She says with a smile. "But if you insist on not marrying me, then at least don't touch me". > say "I will marry you" She nods happily and kisses you passionately on your lips. The two of you embrace each other as you kiss her deeply. It is only after this that you realize that you are actually married.

nineteen999 7y ago
AI has advanced so far it might be able to win the Bulmer-Lytton fiction contest.
blackearl 7y ago
Sounds awesome, it's like reading those curated short stories written by AI. The tech isn't quite there yet, but it makes it all the more hilarious. It's like an alien who's been watching our TV for years comes and tries to write a sitcom.

z3t4 7y ago

I'm too scared Google will charge my credit card just by opening this lab page... I've got a gaming rig, why can't I just run this locally?

Archit3ch 7y ago
Because you don't need a gaming rig, you need a compute rig. Apparently, the model runs on K80s and requires 12GB of GPU RAM. To put that in context, flagship gaming card RTX 2080 Ti "only" has 11GB.
jacobwilliamroy 7y ago
I have an unpriveledged user account named "wilson" designed specifically for situations like this.

csomar 7y ago

I am, personally, amazed by this. The following might be kinda NSFW, though. > You are Jesus, a knight living in the kingdom of Larion. You have a steel longsword and a wooden shield. You are on a quest to defeat the evil dragon of Larion. You've heard he lives up at the north of the kingdom. You set on the path to defeat him and walk into a dark forest. As you enter the forest you see a huge black dragon flying above it's lair. The beast looks very angry as you approach. + suck his d-ck > Your screams are silenced when you feel the head of the dragon's ---- entering your mouth. He begins thrusting inside of you and soon you're filled with his ---. You collapse onto the ground and die Well. That went really more wild than I could have ever expected.

jacobwilliamroy 7y ago
I don't get it. What's NSFW here? Suck his what? Also, yeah I think it's funny how chatbots can generate undesirable or NSFW responses and there's basically no way to stop it. I know GPT-2 can parrot old-timey scientific racism back at me, just like the redditors who trained it!
- csomar 7y ago
  NSFW as don't go loudly reading it in an office.
aasasd 7y ago
> suck his d-ck Thank you for shading my eye from this indignity.
HoveringOrb 7y ago
Still a better love story than Twilight.

YeGoblynQueenne 7y ago

Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly? From what I've seen so far the "game" (if you can even call it that) highlights the weaknesses of GPT-2 much more than its strengths (the model's answers to user actions are random, the story is incoherent, the world is inconsistent). I don't get the feeling it was setup to demonstrate _weakness_ though. I think it was meant to demonstrate strngths. I suppose it's just advertising for the BYU Perception, Control and Cognition Lab, but it sounds awfuly expensive for advertising for an academic group.

qxnqd 7y ago
Why justify it, when it's public money? It's free!
- drewbug 7y ago
  > Brigham Young University is a private research university located in Provo, Utah and owned by The Church of Jesus Christ of Latter-day Saints.
  - deleted 7y ago
    [deleted]
mola 7y ago
Welcome to the hype cycle
Tenoke 7y ago
>Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly? Most popular Games, Movies, etc. cost orders of magnitude more. The value is typically entertainment.
megla_ 7y ago
pseudointellectuallity: the post
7y ago

echelon 7y ago

When will we come full circle and do "edge computing" on our own devices again? I'm getting sick of exorbitant cloud costs and the moat that is forming.

nickbauman 7y ago

IIRC correct bandwidth costs have been essentially flat for the past decade or so. Not coming down.

zaroth 7y ago

So to catch up, AID2 is leveraging a cute offering by Google Colab which will provide any Google account a few hours of free access to a fairly powerful GPU. To use Colab under this arrangement, each person playing the game has to load the model into their personal VM, which is being billed by Google as 6GB of data transfer from GCP into Colab. The obvious questions; 1) Was there a place they could have hosted the model closer to Colab so that they weren’t being charged for egress bandwidth — and also, ideally, so that 6GB of data wasn’t actually being moved very far? 2) The underlying model is 6GB, but I’m curious how much memory is required for an individual user’s world state and how hard it would be to have a single GPU handling multiple user sessions? Presumably it would be possible to multiplex multiple sessions with a single GPU? You would have to serialize the game state, receive the next input, load the prior state, feed the new input through the model, return the resulting text output, and re-serialize the state until the next input comes through. What I don’t know if that’s at all practical based on the amount of data that would have to be serialized? Is the 6GB model data separate and static throughout the game, with an isolated block of data for the current world-state? Or does playing the game fundamentally alter the state of the model, meaning you would have to reload the whole thing just to process the next command?

Akababa 7y ago
Google Drive is actually perfect for this as it lets you mount the drive (presumably as NAS) so you can read from it directly. I set up a notebook for it but unfortunately the shared folder got rate-limited. It still works if you make a copy of the files in your own Google drive though. https://colab.research.google.com/github/Akababa/AIDungeon/blob/patch-2/AIDungeon_2.ipynb https://colab.research.google.com/github/Akababa/AIDungeon/b...
abecedarius 7y ago
The obvious question to me is why must the cost fall on the provider instead of the player? If micropayments were easy enough then this would've been sustainable.
killerstorm 7y ago
"World state" is the text people see on script. It can reside in the browser. So a server can be completely stateless, i.e. it receives text so far (say, 5 KB), applies GPT-2 generate and returns. The problem is that GPT-2 generate seems to be very computationally intensive. As I understand, it actually does number crunching with all these 6 GB of data, so it takes 5-10 seconds even on GPU (K80, at least, is that slow). Is GPU capable of running multiple GPT-2 generate in parallel? No idea. Assuming that a high-end GPU would be able to produce response in 2 seconds, you can only run maybe 10 concurrent sessions per server if you want fast response time.

hombre_fatal 7y ago

I know almost nothing about AI/ML. I can imagine that running it on a single server and responding to requests is intractable because of how the game feeds your quest back into its model. But what would it take to package it as a local application? I really love this game. You can't beat this: https://twitter.com/ptychomancer/status/1203246078989987840 https://twitter.com/ptychomancer/status/1203246078989987840 > $ Give rousing speach to my fellow mud beings > "Mud creatures! Mud creatures! We must unite against this enemy!" > The other mud creatures nod eagerly, and begin chanting, "We will fight! We will fight! We will fight! We will fight! We will..".

sixtypoundhound 7y ago

Speaking as someone who runs a bunch of large websites - out of my own pocket, for profit.... I'm confused. How did we get from 60K users to 10K per day expense? For comparison, I serviced millions of users per month for years from a single virtual server... (granted, that was after making our site super lean for a data & CPU perspective) How much resources is each user consuming?

tiborsaas 7y ago
I guess you don't instantiate a 5Gb image / user. It's the data transfer that costs 30/40c / user.
- rvnx 7y ago
  Cloudflare ? or just one machine at OVH ?
  - rvnx 7y ago
    https://wasabi.com/cloud-storage-pricing/ https://wasabi.com/cloud-storage-pricing/ Wasabi’s pricing model of $.0059 per GB/mo ($5.99 per TB/month) with no additional charges for egress or API requests means you don’t pay to access your data I don't know, could be a couple of dollars only ?

northfoxz 7y ago

Out of curiosity, how much more time will it cost to run this on general CPUs of users computer?

Filligree 7y ago
It takes a couple seconds per reply. GPUs aren't as powerful as everyone would have it, at least not for ML; you'd be lucky to get a 10x speedup.

slacka 7y ago

Could someone explain the appeal in keeping this going long-term? When it was posted here, I played it and read several adventure logs here and on reddit. In every case the story is nonsense. Sure some parts read like something a human would write, but anytime you go beyond a few sentences you can see contractions and lack of flow that good human author would never make. Don't get me wrong, it's a cool demo of how far we have gone beyond Markov chains. Am I missing something or just spoiled from those Infocom games I played as a kid?

buboard 7y ago
GPT2 is trained on huge amounts of text. it contains every book people haven't read, and can more or less stitch pieces together seamlessly. This is exciting from a research perspective but the hype is out of control already. Despite breaking previosu benchmarks, language benchmarks themselves are not great ways to describe language. I get the excitement but this overselling to the general public is soon gonna lead to a trough of disillusionment.
hombre_fatal 7y ago
Man, why so cynical? I don't see how those Infocom games spoiled you since they were limited to whatever a team of writers could come up with. And a well-written, dynamical-feeling CRPG is so rare that we still trumpet the handful that were any good from 20 years ago like Planescape: Torment. Here's an example of how this game is fun: https://twitter.com/ptychomancer/status/1203246078989987840 https://twitter.com/ptychomancer/status/1203246078989987840 It's just fun to play with. > you can see contractions and lack of flow that good human author would never make. That doesn't seem like a sensible goal post. Unless you think the technology is magic, why would you go into this thinking it's going to compete with a master-planned work of fiction by a human writer? I, on the other hand, am inspired by the game. Imagine Crusader Kings 2 (free on Steam btw) where the events are randomly generated by this kind of story-telling technology. Right now it's kind of boring wondering which of the finite human-written events are going to show up. After playing for a while you go from wondering what crazy event will happen next to knowing all of the events and waiting for your favorite ones to show up. We're a ways off from embedding a game in this technology, but I think we are within reach of embedding this technology inside a narrative-driven game. Another example is Dungeons & Dragons. The fun is the sandbox and interacting with the narrative even though the human-driven dynamic storyline is often complete nonsense if you were to read a transcription of what actually happened. This is the second comment I've read so far that seems to gleefully pat itself on the back for wondering how someone could enjoy something else.

TomBombadildoze 7y ago

Someone, probably: People who don't know how their services work are doomed to spend lots of money on them. From the GCS pricing page: Network egress within Google Cloud applies when you move or copy data from one bucket in Cloud Storage to another or when another Google Cloud service accesses data in your bucket. Within the same location (for example, US-EAST1 to US-EAST1 or EU to EU) -- Free From the original tweet (not the linked reply): For reference most of the fees are from transferring from NA to EU and ACAP It's costing them assloads of money because they're moving data between regions. AWS works the same way. Azure is probably also the same but their pricing page is incomprehensible so who knows. Lesson: if you're going to use data in a given location, you need to host data in the same location.

deleted 7y ago
[deleted]
- darkcha0s 7y ago
  Yeah, but your cloud sucks.
  - capn_cabbage 7y ago
    Per the HN Guidelines: "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Hard to say their cloud "sucks" when they offer a solution featuring the very thing being complained about in regards to other cloud services. Or maybe IBM's cloud really is terrible, but it would be hard to know that as an outsider since you failed to provide any substantiating info. If you could expand upon your initial assertion, that would surely help the rest of us see your point and assist you in proving your case.
  - NiceGuy_Ty 7y ago
    What value does this comment add?
    - awillen 7y ago

im3w1l 7y ago

You are Faif, a ranger living in the kingdom of Larion. You have a hunting bow and a quiver of arrows. You spot the deer and are ready to finish your hunt when suddenly you hear a voice calling out to you from behind a tree. > nosaving Saving turned off. > shoot the deer You quickly turn around and see a young man with a long beard standing there. He is dressed in rags and his face is covered by a large bandage. His eyes are red and puffy and he has a large scar across his cheek > shoot the man You fire an arrow at him and it hits him square in the chest. The arrow flies through the air and slams into the ground. You look up and see that the man is lying on the ground motionless. You can't believe what just happened > Check his pockets You pick up the arrow and examine it. It's made of bone and is very sharp. You wonder if you should take it. You could use this to kill someone else. Doesn't seem to understand me too well?

hombre_fatal 7y ago
It's hit or miss. You'll have better results being more specific in your commands because it gives the model more to work off of and let's you crystalize elements of the scene through reaffirmation. "Fire an arrow at the deer" or "Check the pockets of the corpse". I'm often surprised at how well the game can understand me and that it has anything to say in reply at all.

cobookman 7y ago

Shouldn’t this be free egress as traffic is straight to a google product/ colab server or is this being downloaded in the browser? Also cost was before cdn was enabled. This kind of traffic generally costs fractions of a cent per GiB after signing a contract with negotiation. “”” Egress to Google products (such as YouTube, Maps, Drive), whether from a VM in Google Cloud with an external IP address or an internal IP address No charge “””

Jamwinner 7y ago

How does one play this without a Google account?

daniel-thompson 7y ago

Update: the issue has been fixed, apparently: "Should note for anyone who comes and sees this that's no longer how were hosting the model. :) Model is now hosted on a peer to peer torrent network so no more costs for us." https://twitter.com/nickwalton00/status/1204064712394076160 https://twitter.com/nickwalton00/status/1204064712394076160

Akababa 7y ago

I was able to get it working without downloading files by mounting Google Drive, although once the shared folder is rate-limited you need to make a copy in your own Google drive: https://colab.research.google.com/github/Akababa/AIDungeon/blob/patch-2/AIDungeon_2.ipynb https://colab.research.google.com/github/Akababa/AIDungeon/b...