4 ms·

Two that I have encountered: Aspera: a high-performance data transfer suit, used in genetics and the film industry. I wrote a blog post about it a year ago an

by Mizza 7y ago

Two that I have encountered: Aspera: a high-performance data transfer suit, used in genetics and the film industry. I wrote a blog post about it a year ago and still get near-daily emails asking me if there is an open source alternative yet. QUIC may provide a basis for a solution, but it needs to be explored. PDMS: software for planning, modeling and simulating oil and gas plants. Main two commercial suites are AVEVA PDMS and Autocad Plant 3D. AVEVA PDMBS is upwards of $1,000 per seat per month (!), so not feasible for projects which I have in mind (more on that in the new year). I'd imagine that Blender could be used as a starting point for a Free alternative, but I'm not sure. This is a tougher one. [1] https://www.ccdatalab.org/blog/a-desperate-plea-for-a-free-software-alternative-to-aspera https://www.ccdatalab.org/blog/a-desperate-plea-for-a-free-s...

kvz 7y ago
tus.io can saturate network connections reliably and in a resumable way, I wonder what more you would require that makes Aspera stand out as the only way to fly? (honest question, and full disclosure: tus core author)
spectramax 7y ago
Open source projects tend to emerge when there is general access to the purpose of that project. You can't expect an open source project for Ocean-based Wind Turbine simulation or molecular synthesis of Organochloride compounds. They're niche industries that require tremendous amount of upfront investment, in both physical assets to realize the utility of the open-source project, and knowledge-base required to operate the software. On the other hand, you could have a piece of software that requires niche knowledge but its used literally everywhere - for e.g. OpenSSL. Not many people can contribute to it, yet it exists as open source. The point I am trying to make is that OpenSSL and your quantum hardware control software are very different things.
- indigodaddy 7y ago
  Something like Cabinet Vision also too niche? Don't think there are really any serious open source contenders in that space.
- Mizza 7y ago
  I actually respectfully disagree with this. In my travels, I have encountered many situations where very niche, industry-specific software is completely Free and Open Source, and I have encountered situations where commonly-used, widely useful software is proprietary with no Free solution. On the Free side, I think one reason is the copyleft nature of the GPL. Another pattern indicator is how modular the software is - smaller bits are easier to open source than massive projects - and the general inclinations of the programmers, their bosses, and their legal teams. That being said, I don't think it's unreasonable to expect a Free alternative to AutoCAD in 2020. (Pipes 3D is an AutoCAD extention). It's widely used software across numerous industries, much like Photoshop, Illustrator or Audition, but there is no Free alternative that I'm aware of. A particularly annoying thing to me here is that AVEVA was actually originally developed and owned by the British government, but it was privatized (I think under Thatcher.) I think the value created for the tax payer would have been much greater to create free tools and software and make them available to everybody rather than just whatever pittance they got from selling it during a privatization firesale.
  - 7952 7y ago
    I think AutoCAD has an Excel like quality within a lot of industries. It is ridiculously versatile, but often used in a very primitive and risky way. And often you end up with a "drawing" that lacks any of the properties of high quality data. It is used like a really advanced pencil with a thousand nebulous features. That is not a good basis for an open source tool, it is just not well defined enough.
  - safwan 7y ago
    The reason of not having a open source alternative of CAD is not having proper domain knowledge. The people who have the domain knowledge are not taking that intiative that the early stage software engineers took.
  - spectramax 7y ago
    Were these open-source projects result of academic research or someone's dissertation? If so, then yes, many niche packages, for e.g. from CERN are open source even though no one has the ability to go and build their own particle accelerator. Another exception is public works paid by taxes such as Government open source projects. And yet, in commercial industry where I've worked - it is hard to find anything that's useful and can replace internal software because there are no free alternatives. I don't expect these things to be out and open!
whorleater 7y ago
Aspera is among the most frustrating technologies I've been forced to use. I used to work at a video website where we would acquire film files from studios directly, and the amount of times Aspera would fritz out easily justified the time to try and develop and internal alternative. We ended up terminating all the Aspera connections at a service and transferring over direct datagrams for the rest.
safwan 7y ago
Your article is really interesting! I did not know about Aspera before reading it. I did not have any idea that changing protocol will have this amount of benefit in data transmission. Also amused that no one have not solved it in Open Source world.
- Mizza 7y ago
  The worst part is that Aspera is actually a proprietary fork of a Free Software project that IBM purchased and then strong-armed institutions into using until it became an industry standard.
  - trynewideas 7y ago
    FASP, specifically: https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol
    - Mizza 7y ago
      Ha, I wrote that article while I was dealing with all this stuff. FASP is the proprietary fork of UDT.
siscia 7y ago
Tsunami UDP is not a good replacement for aspera?
ablekh 7y ago
Have you tried GridFTP for fast transfer of large data sets?

by Mizza 7y ago

kvz 7y ago

tus.io can saturate network connections reliably and in a resumable way, I wonder what more you would require that makes Aspera stand out as the only way to fly? (honest question, and full disclosure: tus core author)

spectramax 7y ago

Open source projects tend to emerge when there is general access to the purpose of that project. You can't expect an open source project for Ocean-based Wind Turbine simulation or molecular synthesis of Organochloride compounds. They're niche industries that require tremendous amount of upfront investment, in both physical assets to realize the utility of the open-source project, and knowledge-base required to operate the software. On the other hand, you could have a piece of software that requires niche knowledge but its used literally everywhere - for e.g. OpenSSL. Not many people can contribute to it, yet it exists as open source. The point I am trying to make is that OpenSSL and your quantum hardware control software are very different things.

indigodaddy 7y ago
Something like Cabinet Vision also too niche? Don't think there are really any serious open source contenders in that space.
Mizza 7y ago
I actually respectfully disagree with this. In my travels, I have encountered many situations where very niche, industry-specific software is completely Free and Open Source, and I have encountered situations where commonly-used, widely useful software is proprietary with no Free solution. On the Free side, I think one reason is the copyleft nature of the GPL. Another pattern indicator is how modular the software is - smaller bits are easier to open source than massive projects - and the general inclinations of the programmers, their bosses, and their legal teams. That being said, I don't think it's unreasonable to expect a Free alternative to AutoCAD in 2020. (Pipes 3D is an AutoCAD extention). It's widely used software across numerous industries, much like Photoshop, Illustrator or Audition, but there is no Free alternative that I'm aware of. A particularly annoying thing to me here is that AVEVA was actually originally developed and owned by the British government, but it was privatized (I think under Thatcher.) I think the value created for the tax payer would have been much greater to create free tools and software and make them available to everybody rather than just whatever pittance they got from selling it during a privatization firesale.
- 7952 7y ago
  I think AutoCAD has an Excel like quality within a lot of industries. It is ridiculously versatile, but often used in a very primitive and risky way. And often you end up with a "drawing" that lacks any of the properties of high quality data. It is used like a really advanced pencil with a thousand nebulous features. That is not a good basis for an open source tool, it is just not well defined enough.
- safwan 7y ago
  The reason of not having a open source alternative of CAD is not having proper domain knowledge. The people who have the domain knowledge are not taking that intiative that the early stage software engineers took.
- spectramax 7y ago
  Were these open-source projects result of academic research or someone's dissertation? If so, then yes, many niche packages, for e.g. from CERN are open source even though no one has the ability to go and build their own particle accelerator. Another exception is public works paid by taxes such as Government open source projects. And yet, in commercial industry where I've worked - it is hard to find anything that's useful and can replace internal software because there are no free alternatives. I don't expect these things to be out and open!

whorleater 7y ago

Aspera is among the most frustrating technologies I've been forced to use. I used to work at a video website where we would acquire film files from studios directly, and the amount of times Aspera would fritz out easily justified the time to try and develop and internal alternative. We ended up terminating all the Aspera connections at a service and transferring over direct datagrams for the rest.

safwan 7y ago

Your article is really interesting! I did not know about Aspera before reading it. I did not have any idea that changing protocol will have this amount of benefit in data transmission. Also amused that no one have not solved it in Open Source world.

Mizza 7y ago
The worst part is that Aspera is actually a proprietary fork of a Free Software project that IBM purchased and then strong-armed institutions into using until it became an industry standard.
- trynewideas 7y ago
  FASP, specifically: https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol
  - Mizza 7y ago
    Ha, I wrote that article while I was dealing with all this stuff. FASP is the proprietary fork of UDT.

siscia 7y ago

Tsunami UDP is not a good replacement for aspera?

ablekh 7y ago

Have you tried GridFTP for fast transfer of large data sets?