6 ms·

I run a file sharing / content delivery platform called pixeldrain: https://pixeldrain.com https://pixeldrain.com The system serves 4 PB of data to 60 million

I run a file sharing / content delivery platform called pixeldrain: https://pixeldrain.com https://pixeldrain.com The system serves 4 PB of data to 60 million visitors per month. I have served 30 PB and 700 million file views since I started tracking usage somewhere in 2018. I'll go from front to back: - Most of the frontend is plain HTML, CSS and JS. I have started transitioning some pages to Svelte. I like this framework for its speed and simplicity - Cloudflare Analytics to get basic info like which pages are popular and where my users are from - The structure of the website (page wrap, menu, footer, etc) is managed with Go's template system - Constellix for Geo-DNS. This automatically sends users to the server closest to them by doing Geo-IP lookups on the nameservers - The user-facing servers are dedicated 10 Gbps Leaseweb servers, stuffed to the brim with SSDs in RAID6 for caching. Each of these servers cost €1200 per month. The storage servers are from Hetzner's SX line. - The OS is Ubuntu 20.04 server edition. I use Ubuntu over Debian because it ships with TCP BBR - The API is written in plain Go. The only HTTP libraries I use are httprouter for routing and Gorilla Websockets - The storage system is custom built to spread files over multiple servers. I call it pixelstore, it's not open source (yet) - The database is ScyllaDB. I landed on this one after going through multiple other systems with severe bottlenecks. I started with MySQL which was limited to a single location, so other locations had high latency. Then I tried CockroachDB, but it kept hanging under the load no matter how much hardware I threw at it. ScyllaDB is very fast and relatively reliable. - UptimeRobot for monitoring - Mailgun for account e-mail verifications Feel free to ask me more questions :-)

devcriollo 5y ago
very popular service in latam (colombia,peru,mexico,venezuela)
- Fornax96 5y ago
  Definitely. Here's a screenshot from my Cloudflare analytics: https://pixeldrain.com/u/dt92pNgf https://pixeldrain.com/u/dt92pNgf Most of my users are Vietnamese. Unfortunately bandwidth is very expensive there, so I have to serve all that traffic from Europe.
BilalBudhani 5y ago
Thanks for sharing these insight & very impressive stats! Just few questions I have: - What is your total infra cost? A cost split between storage & servers would be an awesome insight - Since how long you have been running this service? - How did you keep up with the infra cost before breaking even?
- Fornax96 5y ago
  Thanks My content delivery servers cost me about €2000 per month. I experiment a lot with different providers to drive cost down here. My storage servers are €800 per month. For that money I get about 500 TB of usable capacity (all my servers run raidz2). And then I spend about €150 on database servers. My Constellix DNS bill is €50 per month, and Mailgun is €5. So about €3000 / month worth of infrastructure. I started working on pixeldrain in 2015 during lunch breaks at school. The original version was a Spring Boot app with MariaDB. I had a lot of performance problems here. My use case doesn't really fit in the CRUD API use case. I need tighter control over my connections and threads. That's why I switched to Go. I run ads to pay for the servers. It's really hard to find advertisers for file sharing services because they don't want to run ads in controversial content, and that's what file sharing sites are mostly used for. I have to resort to working with some really shady companies. Because of this I balance my ads so that they generate just enough revenue to pay for the servers. Eventually I would like to turn some profit though. That's why I'm trying to shift from file sharing to content delivery.
  - wooque 5y ago
    How do you keep yourself motivated to work on a product if it does not turn profit for years?
    - Fornax96 5y ago
      I'm hoping to find a way to make it profitable somehow. There must be something that I can sell to these millions of people who visit my site every day. And I simply like the technical challenges it provides. I like optimizing and scaling systems and designing database schemas. I'm learning tons of stuff this way. If an opportunity provides itself to apply this knowledge I'll have it ready. Pixeldrain as it's running right now could probably do 20 PB / month. But I have systems in mind which could take it to 100 PB / month at little additional cost.
viertaxa 5y ago
Hi! I had never heard of your service, so I googled it and accidentally misspelled "Pixel" as I often do as "Pixle". The first result was a reddit thread and the second seemed to be a full-length rip of Becoming Royal that had some really malicious ads which were clickjacking and covering the content. Just thought you might like to know. :)
- Fornax96 5y ago
  Thanks for letting me know. I suspect my advertising partner is doing some shady business on my site. I have been trying to catch them but I never seem to get the malicious ads myself, even when using a VPN from more eastern or southern nations. Could you maybe send me a screenshot / screen capture of the ads and send them to support@pixeldrain.com? I'd really appreciate that.
  - viertaxa 5y ago
    I sent you an email. Domain is the same as my un here.
- tyingq 5y ago
  That page is the second organic result for "pixledrain" for me. The url is https://pixeldrain.com/u/823R39Sc https://pixeldrain.com/u/823R39Sc And it is full of popups, redirects, etc. And maybe acting differently per-referer.
  - Fornax96 5y ago
    Do you happen to be using an iphone? The malicious ads seem to be targeting iphones specifically. I can't reproduce it myself but I have told my advertiser about it. I have some leverage here since I'm one of their largest publishers.
    - tyingq 5y ago
      Nope, Android. I tried reloading a few times, and the page is normal most of time. So it must be only when a specific ad runs? Edit: Played around with it a bit. I think (not sure), some of the stuff is coming from this in the page source, near the bottom. <script type="text/javascript" src="//optyruntchan.com/400/4189877" data-cfasync="false" async="async"></script> </script>
themulticaster 5y ago
You mention that you have user-facing servers as well as storage servers. So do the user-facing servers act as reverse proxies for the storage servers, or do you simply serve a redirect? I'd expect that the file access patterns are power-law distributions, i.e. recently uploaded files are requested more often than older files. If that's the case, can you use this property for sharding by having hot and cold storage servers? How do you handle users uploading forbidden content? I see from another comment that you ban the usual types of illegal content. But in practice, do you manually review every mail you get on your abuse contact and take appropriate action? What's the most common type of complaint? From a business perspective: How did you grow your site? I imagine the competition must be rough since file hosting is such a "simple" service.
- Fornax96 5y ago
  > So do the user-facing servers act as reverse proxies for the storage servers, or do you simply serve a redirect? Yes, my user-facing servers are proxying the files to the users. > I'd expect that the file access patterns are power-law distributions Exactly. On each server I have a sorted slice in-memory which keeps track of how often files are being requested. The most popular files are cached and files which drop out of the cache are moved to HDD storage servers in Helsinki. This way I can serve 10 Gbps of data to the users while only putting a 1 Gbps load on the storage nodes. > How do you handle users uploading forbidden content? I get a lot of abuse mails, mainly copyright violations. I have a mailserver which is hooked into pixeldrain which scans mails from common copyright offices and automatically blocks the files. For other types of abuse I have a report button on the download page. Users report content which breaks the rules and once the number of reports hits a certain threshold the file is blocked. > From a business perspective: How did you grow your site? Most file sharing sites are really terrible. Like almost unusable. It's pretty easy to beat the competition in UX. I just started using pixeldrain myself on reddit and other forums. In the beginning people complained in the comments about the ergonomics of the site and I listened carefully. Eventually when the kinks were worked out other people started using it too.
  - themulticaster 5y ago
PeterCorless 5y ago
Yeah! Do you want to do a blog for us at ScyllaDB? Would love to interview you. :D Hit me up via email at peter@scylladb.com
- Fornax96 5y ago
  That sounds interesting. I'll send you a mail tomorrow
brundolf 5y ago
> Each of these servers cost €1200 per month I see on Patreon that you're getting $275.43/month from supporters. How does that work out financially?
- AkshatM 5y ago
  From the website: > Pixeldrain is struggling to get by financially. Because anyone can upload anything it's hard to find reputable advertisers who want to advertise on pixeldrain. Every month the ad revenue just barely covers the bandwidth costs. If this continues I will have to resort to adding more shady ads, or reducing the file size and bandwidth limits. That's not something I would like to do.
- AkshatM 5y ago
  From the website: > Pixeldrain is struggling to get by financially. Because anyone can upload anything it's hard to find reputable advertisers who want to advertise on pixeldrain. Every month the ad revenue just barely covers the bandwidth costs. If this continues I will have to resort to adding more shady ads, or reducing the file size and bandwidth limits. That's not something I would like to do
- Fornax96 5y ago
  The vast majority of my income is from ads. It's a 10:1 ratio. Patreon definitely helps, and I hope that it will get so far that I have to rely less on ads to keep the site running. The nice thing about Patreon is that they do my taxes for me. When I receive my money the taxes have already been paid. I can put it in my accounting software as tax free money. Patreon's fees are high though, so if something else comes along I'll give that a shot too.
sergiotapia 5y ago
Curious to learn more about CockroachDB's problems you had. They market themselves as a turn-key solution to globally distributed databases, it's worrying that you had a problem with that very use case. Thanks for the additional details!
- Fornax96 5y ago
  Okay this is just my understanding of the issue. I'm not sure if it's 100% correct. Cockroachdb slices up its tables in 16 MiB shards. These shards are distributed over three nodes, one master and two slaves. The master gets chosen with the raft protocol. The master is the only server which can write to the shard. For a write to happen all three replicas need to be synchronized, they do this by locking the entire shard whenever a query comes in that writes to a row in the shard. The problem is that if the latency between the master and the slave node is too large, you are severely limited in how many queries can be executed. For example if you have one shard in the US and one in Europe there might be a latency of 150 - 200ms between them. This means you can only execute 5 queries per second. Any more requests that come in are queued. After a while the queue gets so long that incoming queries have to be dropped. This implementation is great for consistency, but if you frequently need to run update queries on your rows it's not the right solution. You need eventual consistency like Scylla and Cassandra provide.
  - sergiotapia 5y ago
    Thank you for sharing!
  - PeterCorless 5y ago
    Good summation. Exactly. They are solving for different things. Cockroach is solving for consistency of a distributed database. Scylla is solving for performance of a distributed database. For interested readers more can be found here: https://www.scylladb.com/2021/01/21/cockroachdb-vs-scylla-benchmark/ https://www.scylladb.com/2021/01/21/cockroachdb-vs-scylla-be...
kadomony 5y ago
What does your revenue look like? It sounds like such a simple service that can basically maintain itself.

devcriollo 5y ago

very popular service in latam (colombia,peru,mexico,venezuela)

Fornax96 5y ago
Definitely. Here's a screenshot from my Cloudflare analytics: https://pixeldrain.com/u/dt92pNgf https://pixeldrain.com/u/dt92pNgf Most of my users are Vietnamese. Unfortunately bandwidth is very expensive there, so I have to serve all that traffic from Europe.

BilalBudhani 5y ago

Thanks for sharing these insight & very impressive stats! Just few questions I have: - What is your total infra cost? A cost split between storage & servers would be an awesome insight - Since how long you have been running this service? - How did you keep up with the infra cost before breaking even?

Fornax96 5y ago
Thanks My content delivery servers cost me about €2000 per month. I experiment a lot with different providers to drive cost down here. My storage servers are €800 per month. For that money I get about 500 TB of usable capacity (all my servers run raidz2). And then I spend about €150 on database servers. My Constellix DNS bill is €50 per month, and Mailgun is €5. So about €3000 / month worth of infrastructure. I started working on pixeldrain in 2015 during lunch breaks at school. The original version was a Spring Boot app with MariaDB. I had a lot of performance problems here. My use case doesn't really fit in the CRUD API use case. I need tighter control over my connections and threads. That's why I switched to Go. I run ads to pay for the servers. It's really hard to find advertisers for file sharing services because they don't want to run ads in controversial content, and that's what file sharing sites are mostly used for. I have to resort to working with some really shady companies. Because of this I balance my ads so that they generate just enough revenue to pay for the servers. Eventually I would like to turn some profit though. That's why I'm trying to shift from file sharing to content delivery.
- wooque 5y ago
  How do you keep yourself motivated to work on a product if it does not turn profit for years?
  - Fornax96 5y ago
    I'm hoping to find a way to make it profitable somehow. There must be something that I can sell to these millions of people who visit my site every day. And I simply like the technical challenges it provides. I like optimizing and scaling systems and designing database schemas. I'm learning tons of stuff this way. If an opportunity provides itself to apply this knowledge I'll have it ready. Pixeldrain as it's running right now could probably do 20 PB / month. But I have systems in mind which could take it to 100 PB / month at little additional cost.

viertaxa 5y ago

Hi! I had never heard of your service, so I googled it and accidentally misspelled "Pixel" as I often do as "Pixle". The first result was a reddit thread and the second seemed to be a full-length rip of Becoming Royal that had some really malicious ads which were clickjacking and covering the content. Just thought you might like to know. :)

Fornax96 5y ago
Thanks for letting me know. I suspect my advertising partner is doing some shady business on my site. I have been trying to catch them but I never seem to get the malicious ads myself, even when using a VPN from more eastern or southern nations. Could you maybe send me a screenshot / screen capture of the ads and send them to support@pixeldrain.com? I'd really appreciate that.
- viertaxa 5y ago
  I sent you an email. Domain is the same as my un here.
tyingq 5y ago
That page is the second organic result for "pixledrain" for me. The url is https://pixeldrain.com/u/823R39Sc https://pixeldrain.com/u/823R39Sc And it is full of popups, redirects, etc. And maybe acting differently per-referer.
- Fornax96 5y ago
  Do you happen to be using an iphone? The malicious ads seem to be targeting iphones specifically. I can't reproduce it myself but I have told my advertiser about it. I have some leverage here since I'm one of their largest publishers.
  - tyingq 5y ago
    Nope, Android. I tried reloading a few times, and the page is normal most of time. So it must be only when a specific ad runs? Edit: Played around with it a bit. I think (not sure), some of the stuff is coming from this in the page source, near the bottom. <script type="text/javascript" src="//optyruntchan.com/400/4189877" data-cfasync="false" async="async"></script> </script>

themulticaster 5y ago

You mention that you have user-facing servers as well as storage servers. So do the user-facing servers act as reverse proxies for the storage servers, or do you simply serve a redirect? I'd expect that the file access patterns are power-law distributions, i.e. recently uploaded files are requested more often than older files. If that's the case, can you use this property for sharding by having hot and cold storage servers? How do you handle users uploading forbidden content? I see from another comment that you ban the usual types of illegal content. But in practice, do you manually review every mail you get on your abuse contact and take appropriate action? What's the most common type of complaint? From a business perspective: How did you grow your site? I imagine the competition must be rough since file hosting is such a "simple" service.

Fornax96 5y ago
> So do the user-facing servers act as reverse proxies for the storage servers, or do you simply serve a redirect? Yes, my user-facing servers are proxying the files to the users. > I'd expect that the file access patterns are power-law distributions Exactly. On each server I have a sorted slice in-memory which keeps track of how often files are being requested. The most popular files are cached and files which drop out of the cache are moved to HDD storage servers in Helsinki. This way I can serve 10 Gbps of data to the users while only putting a 1 Gbps load on the storage nodes. > How do you handle users uploading forbidden content? I get a lot of abuse mails, mainly copyright violations. I have a mailserver which is hooked into pixeldrain which scans mails from common copyright offices and automatically blocks the files. For other types of abuse I have a report button on the download page. Users report content which breaks the rules and once the number of reports hits a certain threshold the file is blocked. > From a business perspective: How did you grow your site? Most file sharing sites are really terrible. Like almost unusable. It's pretty easy to beat the competition in UX. I just started using pixeldrain myself on reddit and other forums. In the beginning people complained in the comments about the ergonomics of the site and I listened carefully. Eventually when the kinks were worked out other people started using it too.
- themulticaster 5y ago

PeterCorless 5y ago

Yeah! Do you want to do a blog for us at ScyllaDB? Would love to interview you. :D Hit me up via email at peter@scylladb.com

Fornax96 5y ago
That sounds interesting. I'll send you a mail tomorrow

brundolf 5y ago

> Each of these servers cost €1200 per month I see on Patreon that you're getting $275.43/month from supporters. How does that work out financially?

AkshatM 5y ago
From the website: > Pixeldrain is struggling to get by financially. Because anyone can upload anything it's hard to find reputable advertisers who want to advertise on pixeldrain. Every month the ad revenue just barely covers the bandwidth costs. If this continues I will have to resort to adding more shady ads, or reducing the file size and bandwidth limits. That's not something I would like to do.
AkshatM 5y ago
From the website: > Pixeldrain is struggling to get by financially. Because anyone can upload anything it's hard to find reputable advertisers who want to advertise on pixeldrain. Every month the ad revenue just barely covers the bandwidth costs. If this continues I will have to resort to adding more shady ads, or reducing the file size and bandwidth limits. That's not something I would like to do
Fornax96 5y ago
The vast majority of my income is from ads. It's a 10:1 ratio. Patreon definitely helps, and I hope that it will get so far that I have to rely less on ads to keep the site running. The nice thing about Patreon is that they do my taxes for me. When I receive my money the taxes have already been paid. I can put it in my accounting software as tax free money. Patreon's fees are high though, so if something else comes along I'll give that a shot too.

sergiotapia 5y ago

Curious to learn more about CockroachDB's problems you had. They market themselves as a turn-key solution to globally distributed databases, it's worrying that you had a problem with that very use case. Thanks for the additional details!

Fornax96 5y ago
Okay this is just my understanding of the issue. I'm not sure if it's 100% correct. Cockroachdb slices up its tables in 16 MiB shards. These shards are distributed over three nodes, one master and two slaves. The master gets chosen with the raft protocol. The master is the only server which can write to the shard. For a write to happen all three replicas need to be synchronized, they do this by locking the entire shard whenever a query comes in that writes to a row in the shard. The problem is that if the latency between the master and the slave node is too large, you are severely limited in how many queries can be executed. For example if you have one shard in the US and one in Europe there might be a latency of 150 - 200ms between them. This means you can only execute 5 queries per second. Any more requests that come in are queued. After a while the queue gets so long that incoming queries have to be dropped. This implementation is great for consistency, but if you frequently need to run update queries on your rows it's not the right solution. You need eventual consistency like Scylla and Cassandra provide.
- sergiotapia 5y ago
  Thank you for sharing!
- PeterCorless 5y ago
  Good summation. Exactly. They are solving for different things. Cockroach is solving for consistency of a distributed database. Scylla is solving for performance of a distributed database. For interested readers more can be found here: https://www.scylladb.com/2021/01/21/cockroachdb-vs-scylla-benchmark/ https://www.scylladb.com/2021/01/21/cockroachdb-vs-scylla-be...

kadomony 5y ago

What does your revenue look like? It sounds like such a simple service that can basically maintain itself.