9 ms·

Show HN: Pixeldrain – File sharing service optimized for speed and ease of use

56 points by Fornax96 5y ago

Fornax96 5y ago
Pixeldrain is a project which I started working on during the lunch breaks at school around 6 years ago. Originally it was intended as a screenshotting tool for Linux. Eventually I realized I had more fun working on the backend than the desktop app, so I abandoned the app and opened up the site for all file types instead of just images. Both the backend and frontend are optimized to be as fast as I can possibly make it. Recently I also responded to a question on HN asking about tech stacks, you can find that one here: https://news.ycombinator.com/item?id=28300657 https://news.ycombinator.com/item?id=28300657. Following the success of that post I thought I should show my project to everyone on HN.
yewenjie 5y ago
How can you provide 500GB of free storage to users? Just curious, do the ads cover that much?
- Fornax96 5y ago
  It's not really meant as cloud storage, but more as a catchnet to prevent abuse. If you're not paying for pixeldrain your files will always expire 30 days after the last time they're viewed. So even if someone uploads 500 GB the chance is very slim that it will still be there next month. As long as the files stay up they're generating ad revenue and I don't need to remove them.
  - nikolay 5y ago
    Oh, really? You didn't know how people used Flickr to store terabytes of data in "images"?
    - Fornax96 5y ago
      Bandwidth costs 10 times as much as storage, so I'm more concerned about that :-) If it becomes a problem I'll do something about it. The storage limit is a recent thing anyway. The site operated 6 years without any limit on storage space.
      - nikolay 5y ago
        Not really. Reliable storage is very expensive!
feross 5y ago
Nice project. I like the text file and hotlinking features! Unfortunately supporting hotlinking makes end-to-end encryption impossible. Also, the inclusion of third-party tracker/ad scripts and the discussion of possibly adding “more shady ads” is concerning. For an E2EE alternative with no third party scripts, see https://wormhole.app https://wormhole.app (Disclosure: I built this)
- Fornax96 5y ago
  Oh yea it's probably not a good idea to threaten users with "more shady ads" on the home page.. I'll remove that. Pixeldrain focuses more on distibuting files to large audiences than one-to-one file transfer. That's why it's not focused on encryption / privacy as much, everything is supposed to be public anyway. I like Wormhole a lot. If I hadn't built my own alternative I would probably be using it regularly :-)
- bil7 5y ago
  oh nice, this looks awesome. Bookmarked
aranjedeath 5y ago
Next please make it more readable. Section 3 of the homepage has 3 different flavors of dark purple. I can barely read it.
- Fornax96 5y ago
  Do you mean the buttons for sharing files? Those are disabled until you have uploaded something. After uploading they become active and the contrast becomes much higher.
jeroenhd 5y ago
I can't reach the page, possibly due to HN's hug of death. An archived version of a non-functional version of the home page is here though: https://web.archive.org/web/20210911035325/https://pixeldrain.com/ https://web.archive.org/web/20210911035325/https://pixeldrai...
- Fornax96 5y ago
  My hosting provider just decided to turn my server off for no apparent reason. I have no idea why yet, waiting on an answer from support. The timing for this is the absolute worst possible... Needless to say I'm going to be looking for alternatives again. If anyone knows of good hosting companies with unmetered 10 Gbps let me know.
  - jeroenhd 5y ago
    FYI, it seems to be back up now! I did notice one potential flaw in the way your API/website works, though, which is that raw access to the uploaded files is enabled regardless of content type. This allows people to use your website to host scripts and other malicious files that they can reuse to execute XSS attacks against other websites. This is mostly a problem for the other website (which would lack proper security headers) but it could lead to Google blacklisting your website for "hosting malware". Some phone scammers leverage websites like yours that try to provide a simple, cheap service to host their fake virus warnings and such. I'd personally recommend against allowing text content to be directly accessible with a single GET request from another source. You've set up some protections against this (rate limiting mode) but that could be circumvented by pointing a browser at the file viewer and placing something heavy on the F5 key. I'd personally also check the origin/referrer headers for files that aren't images or video and block their direct inclusion in some way.
    - Fornax96 5y ago
      Views are unique per IP address (/48 subnet for IPv6) so continuously refreshing the page wouldn't work. I had problems like this in the past, but rate limiting mode seems to have solved it. I have had little abuse complaints in that regard the last few years. But thanks for the suggestion, I'll look into it.
leppr 5y ago
I'm curious what was the biggest difficulty you encountered when building this? If I had to guess I would imagine it to be the economics behind finding suitable hosting providers and pricing rather than something purely technical, but having never built anything like it this could be completely off-base. Maybe moderation of user content will become the big one.
- Fornax96 5y ago
  Scaling has been tricky. Finding a database system which can handle the load was the hardest part. A single Postgres instance buckled quite early on. Later I tried Cockroachdb but it also was not up for the task. Now I'm running ScyllaDB and it's handling tens of thousands of queries per seconds like a champ. Content moderation is already a big problem. It's what I spend most of my time on nowadays. There is a lot of crazy shit getting uploaded. Abuse reports are flying in from all corners of the internet. I'm working on automated systems for processing them. The most annoying part is that a lot of copyright management firms don't sign their e-mails with DKIM, so I can't verify that the sender address was not spoofed. This prevents me from fully automating the e-mail responses.
mtlynch 5y ago
I discovered PixelDrain when I was involved in open source distributed storage projects in 2015-2017. Glad to see it still going strong! Are you still using any kind of decentralized storage technology on the backend?
- Fornax96 5y ago
  Hey Michael, good to hear from you here! I still have a Sia daemon running but it's not doing much. When the site still ran on Sia I had a lot of issues with the scalability of the Sia daemon. While it's still getting better, it's still not as scalable as I'd like. The team has shifted their focus to building a new decentralized internet and the original Sia software has not seen much development since. I might try their new Skynet daemon in the future to see if it scales any better. I am still upholding your tradition of benchmarking new Sia releases from time to time. If you're curious, here are the latest results: https://siastats.info/benchmarking https://siastats.info/benchmarking
  - mtlynch 5y ago
    Oh, interesting. I knew about Skynet, but I figured they'd still need to develop Sia. Looking at the repo now, I see what you mean. It looks like development on the repo stopped earlier this year.[0] I didn't realize the benchmarks were still going. That's cool! Thanks for carrying the torch. It's interesting to see how Sia's metrics have evolved over time. [0] https://gitlab.com/NebulousLabs/Sia https://gitlab.com/NebulousLabs/Sia
    - Fornax96 5y ago
      They moved repos again. The Sia daemon is now here: https://github.com/SiaFoundation/siad https://github.com/SiaFoundation/siad And Skynet was forked into a different repo: https://gitlab.com/SkynetLabs/skyd https://gitlab.com/SkynetLabs/skyd

56 points by Fornax96 5y ago

Pixeldrain is a project which I started working on during the lunch breaks at school around 6 years ago. Originally it was intended as a screenshotting tool for Linux. Eventually I realized I had more fun working on the backend than the desktop app, so I abandoned the app and opened up the site for all file types instead of just images. Both the backend and frontend are optimized to be as fast as I can possibly make it. Recently I also responded to a question on HN asking about tech stacks, you can find that one here: https://news.ycombinator.com/item?id=28300657 https://news.ycombinator.com/item?id=28300657. Following the success of that post I thought I should show my project to everyone on HN.

yewenjie 5y ago

How can you provide 500GB of free storage to users? Just curious, do the ads cover that much?

Fornax96 5y ago
It's not really meant as cloud storage, but more as a catchnet to prevent abuse. If you're not paying for pixeldrain your files will always expire 30 days after the last time they're viewed. So even if someone uploads 500 GB the chance is very slim that it will still be there next month. As long as the files stay up they're generating ad revenue and I don't need to remove them.
- nikolay 5y ago
  Oh, really? You didn't know how people used Flickr to store terabytes of data in "images"?
  - Fornax96 5y ago
    Bandwidth costs 10 times as much as storage, so I'm more concerned about that :-) If it becomes a problem I'll do something about it. The storage limit is a recent thing anyway. The site operated 6 years without any limit on storage space.
    - nikolay 5y ago
      Not really. Reliable storage is very expensive!

feross 5y ago

Nice project. I like the text file and hotlinking features! Unfortunately supporting hotlinking makes end-to-end encryption impossible. Also, the inclusion of third-party tracker/ad scripts and the discussion of possibly adding “more shady ads” is concerning. For an E2EE alternative with no third party scripts, see https://wormhole.app https://wormhole.app (Disclosure: I built this)

Fornax96 5y ago
Oh yea it's probably not a good idea to threaten users with "more shady ads" on the home page.. I'll remove that. Pixeldrain focuses more on distibuting files to large audiences than one-to-one file transfer. That's why it's not focused on encryption / privacy as much, everything is supposed to be public anyway. I like Wormhole a lot. If I hadn't built my own alternative I would probably be using it regularly :-)
bil7 5y ago
oh nice, this looks awesome. Bookmarked

aranjedeath 5y ago

Next please make it more readable. Section 3 of the homepage has 3 different flavors of dark purple. I can barely read it.

Fornax96 5y ago
Do you mean the buttons for sharing files? Those are disabled until you have uploaded something. After uploading they become active and the contrast becomes much higher.

jeroenhd 5y ago

I can't reach the page, possibly due to HN's hug of death. An archived version of a non-functional version of the home page is here though: https://web.archive.org/web/20210911035325/https://pixeldrain.com/ https://web.archive.org/web/20210911035325/https://pixeldrai...

Fornax96 5y ago
My hosting provider just decided to turn my server off for no apparent reason. I have no idea why yet, waiting on an answer from support. The timing for this is the absolute worst possible... Needless to say I'm going to be looking for alternatives again. If anyone knows of good hosting companies with unmetered 10 Gbps let me know.
- jeroenhd 5y ago
  FYI, it seems to be back up now! I did notice one potential flaw in the way your API/website works, though, which is that raw access to the uploaded files is enabled regardless of content type. This allows people to use your website to host scripts and other malicious files that they can reuse to execute XSS attacks against other websites. This is mostly a problem for the other website (which would lack proper security headers) but it could lead to Google blacklisting your website for "hosting malware". Some phone scammers leverage websites like yours that try to provide a simple, cheap service to host their fake virus warnings and such. I'd personally recommend against allowing text content to be directly accessible with a single GET request from another source. You've set up some protections against this (rate limiting mode) but that could be circumvented by pointing a browser at the file viewer and placing something heavy on the F5 key. I'd personally also check the origin/referrer headers for files that aren't images or video and block their direct inclusion in some way.
  - Fornax96 5y ago
    Views are unique per IP address (/48 subnet for IPv6) so continuously refreshing the page wouldn't work. I had problems like this in the past, but rate limiting mode seems to have solved it. I have had little abuse complaints in that regard the last few years. But thanks for the suggestion, I'll look into it.

leppr 5y ago

I'm curious what was the biggest difficulty you encountered when building this? If I had to guess I would imagine it to be the economics behind finding suitable hosting providers and pricing rather than something purely technical, but having never built anything like it this could be completely off-base. Maybe moderation of user content will become the big one.

Fornax96 5y ago
Scaling has been tricky. Finding a database system which can handle the load was the hardest part. A single Postgres instance buckled quite early on. Later I tried Cockroachdb but it also was not up for the task. Now I'm running ScyllaDB and it's handling tens of thousands of queries per seconds like a champ. Content moderation is already a big problem. It's what I spend most of my time on nowadays. There is a lot of crazy shit getting uploaded. Abuse reports are flying in from all corners of the internet. I'm working on automated systems for processing them. The most annoying part is that a lot of copyright management firms don't sign their e-mails with DKIM, so I can't verify that the sender address was not spoofed. This prevents me from fully automating the e-mail responses.

mtlynch 5y ago

I discovered PixelDrain when I was involved in open source distributed storage projects in 2015-2017. Glad to see it still going strong! Are you still using any kind of decentralized storage technology on the backend?

Fornax96 5y ago
Hey Michael, good to hear from you here! I still have a Sia daemon running but it's not doing much. When the site still ran on Sia I had a lot of issues with the scalability of the Sia daemon. While it's still getting better, it's still not as scalable as I'd like. The team has shifted their focus to building a new decentralized internet and the original Sia software has not seen much development since. I might try their new Skynet daemon in the future to see if it scales any better. I am still upholding your tradition of benchmarking new Sia releases from time to time. If you're curious, here are the latest results: https://siastats.info/benchmarking https://siastats.info/benchmarking
- mtlynch 5y ago
  Oh, interesting. I knew about Skynet, but I figured they'd still need to develop Sia. Looking at the repo now, I see what you mean. It looks like development on the repo stopped earlier this year.[0] I didn't realize the benchmarks were still going. That's cool! Thanks for carrying the torch. It's interesting to see how Sia's metrics have evolved over time. [0] https://gitlab.com/NebulousLabs/Sia https://gitlab.com/NebulousLabs/Sia
  - Fornax96 5y ago
    They moved repos again. The Sia daemon is now here: https://github.com/SiaFoundation/siad https://github.com/SiaFoundation/siad And Skynet was forked into a different repo: https://gitlab.com/SkynetLabs/skyd https://gitlab.com/SkynetLabs/skyd