5 ms·

Please consider producing archives in WARC format, and either donating captures of public pages to the Internet Archive (and other interested archives), or supp

Please consider producing archives in WARC format, and either donating captures of public pages to the Internet Archive (and other interested archives), or supporting ways for users to download their own archives in that format for them to donate them themselves and use in systems like Webrecorder. (Note that a download of just page content and assets isn't enough; WARC stores headers, etc., also.)

ernsheong 9y ago
Thanks for the comment. Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. If I have a better understanding of WARC maybe that can be done, but I make no promises.
- bravura 9y ago
  I'm not sure why this is getting downvoted. WARC tooling is pretty terrible. I hack data and text for a living and working with WARC was extremely difficult compared to the simplicity of what I was trying to do. Something that would have taken me five minutes (extract body text) took me about 60 - 90 minutes because the first few libraries I tried didn't work easily.
  - toomuchtodo 9y ago
    Both wget and it’s python spiritual successor wpull support WARC creation natively.
  - gooseus 9y ago
    I don't think it's getting downvoted because WARC is actually easy, I think it's because the creator says they can't be bothered to figure out the web archiving standard before asking people to start giving them money for their web archiving product. The fact that WARC tooling is terrible is the opportunity this person should be leveraging to create value, imo.
    - j_s 9y ago
      I will reference this thread when I need an example of minimum viable product catching flak. Putting off WARC makes financial sense.
      - gooseus 9y ago
        The response wasn't: "I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. If it's an essential feature to make this product valuable then I will definitely make it a priority." If it was "I felt overwhelmed...", "...maybe..." and "...no promises". I appreciate the honesty, but it doesn't inspire confidence and I think the flak is more a reaction to the response regarding WARC than the lack of WARC itself.
        j_s 9y ago
        True. My take-away: don't overshare motivations for decisions in response to MVP feedback. Instead: respond to criticism with a minimal deflection stating "it's on the roadmap".
        jasonlotito 9y ago
        I feel both your suggested response and the given response amount to the same thing. I mean, looking at what you said and what they said side by side, and how I read it: > Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. > I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. Both mention looking at using WARC, and both mention it being complicated. Both say it's not yet available. > If I have a better understanding of WARC maybe that can be done, but I make no promises. > If it's an essential feature to make this product valuable then I will definitely make it a priority. Both offer up the potential for supporting WARC. Neither promise its delivery. The former provides a bit more justification as to the condition for its inclusion (technical capabilities), while the later hides this fact. > I appreciate the honesty, Honesty was the only difference between the response you hoped for and the response provided.
  - xfer 9y ago
    https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio - python library. Tooling is not the best, which is precisely why anybody would bother paying for their service. You can do lots of fancy stuff with warc and cdx index. Check out WAIL(https://github.com/N0taN3rd/wail https://github.com/N0taN3rd/wail), it works great, but it's slow and resource heavy. EDIT: You might also want to try out webrecorder self-hosted. Both use the same library, pywb for archiving.
- leepowers 9y ago
  Do you have an API? Even if WARC isn't something you feel comfortable tackling an API or webhooks would enable other enterprising developers to add this feature. Some commenters are knocking PageDash for being a cloud service, and that's understandable. You'll have an easier time on-boarding HN users if you make it easy for them to access and control their data.
  - ernsheong 9y ago
    This is something some folks have been asking. I think it's still early days, but it's something for a bit later.
jaytaylor 9y ago
http://fileformats.archiveteam.org/wiki/WARC http://fileformats.archiveteam.org/wiki/WARC Never heard of this until now, it sounds exceptionally pragmatic and good! I also found these WARC tools made by the folks at Internet Archive, certainly interesting: https://github.com/internetarchive/warctools https://github.com/internetarchive/warctools
AdieuToLogic 9y ago
For those who had not heard of the WARC format before now (which included myself), I believe this is the official specification: http://archive-access.sourceforge.net/warc/warc_file_format-0.9.html http://archive-access.sourceforge.net/warc/warc_file_format-... As indicated by the US Library of Congress[0]. 0 - https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...
tomc1985 9y ago
Wait, you can't download archived pages? ANOTHER ----id ----ing cloud service trying to replace files and programs with some BS pricing scheme? Seriously are the "entrepreneurs" of HN even trying? How pathetic that seemingly everything on this site is someone's jobs program?
- ernsheong 9y ago
  Downloading is definitely on the roadmap. Gotta start somewhere.
  - tomc1985 9y ago
    Fair enough, my rage is misplaced. It is hard not to get worked up when it seems like everyone is out to replace the old ways with these newer, more dis-empowering "cloud" alternatives

ernsheong 9y ago

Thanks for the comment. Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. If I have a better understanding of WARC maybe that can be done, but I make no promises.

bravura 9y ago
I'm not sure why this is getting downvoted. WARC tooling is pretty terrible. I hack data and text for a living and working with WARC was extremely difficult compared to the simplicity of what I was trying to do. Something that would have taken me five minutes (extract body text) took me about 60 - 90 minutes because the first few libraries I tried didn't work easily.
- toomuchtodo 9y ago
  Both wget and it’s python spiritual successor wpull support WARC creation natively.
- gooseus 9y ago
  I don't think it's getting downvoted because WARC is actually easy, I think it's because the creator says they can't be bothered to figure out the web archiving standard before asking people to start giving them money for their web archiving product. The fact that WARC tooling is terrible is the opportunity this person should be leveraging to create value, imo.
  - j_s 9y ago
    I will reference this thread when I need an example of minimum viable product catching flak. Putting off WARC makes financial sense.
    - gooseus 9y ago
      The response wasn't: "I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. If it's an essential feature to make this product valuable then I will definitely make it a priority." If it was "I felt overwhelmed...", "...maybe..." and "...no promises". I appreciate the honesty, but it doesn't inspire confidence and I think the flak is more a reaction to the response regarding WARC than the lack of WARC itself.
      - j_s 9y ago
        True. My take-away: don't overshare motivations for decisions in response to MVP feedback. Instead: respond to criticism with a minimal deflection stating "it's on the roadmap".
      - jasonlotito 9y ago
        I feel both your suggested response and the given response amount to the same thing. I mean, looking at what you said and what they said side by side, and how I read it: > Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. > I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. Both mention looking at using WARC, and both mention it being complicated. Both say it's not yet available. > If I have a better understanding of WARC maybe that can be done, but I make no promises. > If it's an essential feature to make this product valuable then I will definitely make it a priority. Both offer up the potential for supporting WARC. Neither promise its delivery. The former provides a bit more justification as to the condition for its inclusion (technical capabilities), while the later hides this fact. > I appreciate the honesty, Honesty was the only difference between the response you hoped for and the response provided.
- xfer 9y ago
  https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio - python library. Tooling is not the best, which is precisely why anybody would bother paying for their service. You can do lots of fancy stuff with warc and cdx index. Check out WAIL(https://github.com/N0taN3rd/wail https://github.com/N0taN3rd/wail), it works great, but it's slow and resource heavy. EDIT: You might also want to try out webrecorder self-hosted. Both use the same library, pywb for archiving.
leepowers 9y ago
Do you have an API? Even if WARC isn't something you feel comfortable tackling an API or webhooks would enable other enterprising developers to add this feature. Some commenters are knocking PageDash for being a cloud service, and that's understandable. You'll have an easier time on-boarding HN users if you make it easy for them to access and control their data.
- ernsheong 9y ago
  This is something some folks have been asking. I think it's still early days, but it's something for a bit later.

jaytaylor 9y ago

http://fileformats.archiveteam.org/wiki/WARC http://fileformats.archiveteam.org/wiki/WARC Never heard of this until now, it sounds exceptionally pragmatic and good! I also found these WARC tools made by the folks at Internet Archive, certainly interesting: https://github.com/internetarchive/warctools https://github.com/internetarchive/warctools

AdieuToLogic 9y ago

For those who had not heard of the WARC format before now (which included myself), I believe this is the official specification: http://archive-access.sourceforge.net/warc/warc_file_format-0.9.html http://archive-access.sourceforge.net/warc/warc_file_format-... As indicated by the US Library of Congress[0]. 0 - https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...

tomc1985 9y ago

Wait, you can't download archived pages? ANOTHER ----id ----ing cloud service trying to replace files and programs with some BS pricing scheme? Seriously are the "entrepreneurs" of HN even trying? How pathetic that seemingly everything on this site is someone's jobs program?

ernsheong 9y ago
Downloading is definitely on the roadmap. Gotta start somewhere.
- tomc1985 9y ago
  Fair enough, my rage is misplaced. It is hard not to get worked up when it seems like everyone is out to replace the old ways with these newer, more dis-empowering "cloud" alternatives