5 ms·

Thanks for the comment. Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. If I have a bet

Thanks for the comment. Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. If I have a better understanding of WARC maybe that can be done, but I make no promises.

bravura 9y ago
I'm not sure why this is getting downvoted. WARC tooling is pretty terrible. I hack data and text for a living and working with WARC was extremely difficult compared to the simplicity of what I was trying to do. Something that would have taken me five minutes (extract body text) took me about 60 - 90 minutes because the first few libraries I tried didn't work easily.
- toomuchtodo 9y ago
  Both wget and it’s python spiritual successor wpull support WARC creation natively.
- gooseus 9y ago
  I don't think it's getting downvoted because WARC is actually easy, I think it's because the creator says they can't be bothered to figure out the web archiving standard before asking people to start giving them money for their web archiving product. The fact that WARC tooling is terrible is the opportunity this person should be leveraging to create value, imo.
  - j_s 9y ago
    I will reference this thread when I need an example of minimum viable product catching flak. Putting off WARC makes financial sense.
    - gooseus 9y ago
      The response wasn't: "I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. If it's an essential feature to make this product valuable then I will definitely make it a priority." If it was "I felt overwhelmed...", "...maybe..." and "...no promises". I appreciate the honesty, but it doesn't inspire confidence and I think the flak is more a reaction to the response regarding WARC than the lack of WARC itself.
      - j_s 9y ago
        True. My take-away: don't overshare motivations for decisions in response to MVP feedback. Instead: respond to criticism with a minimal deflection stating "it's on the roadmap".
      - jasonlotito 9y ago
        I feel both your suggested response and the given response amount to the same thing. I mean, looking at what you said and what they said side by side, and how I read it: > Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. > I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. Both mention looking at using WARC, and both mention it being complicated. Both say it's not yet available. > If I have a better understanding of WARC maybe that can be done, but I make no promises. > If it's an essential feature to make this product valuable then I will definitely make it a priority. Both offer up the potential for supporting WARC. Neither promise its delivery. The former provides a bit more justification as to the condition for its inclusion (technical capabilities), while the later hides this fact. > I appreciate the honesty, Honesty was the only difference between the response you hoped for and the response provided.
        gooseus 9y ago
        Really?? The tone and focus of the two statements are entirely different and communicate completely different impressions of the creators relationship to the question/criticism. I mean I don't really want to get pedantic here, but... I will anyways, weird mood I suppose: My statement implies an understanding of the value of WARC and an intention to implement the feature, but tempered by a desire to understand the needs of the customer before taking on the task. Their response indicates a lack of understanding of the technology (some of the honesty that I appreciate, fyi) and offers a non-commitment to implementing it, even IF they were able to understand it. I'll admit my impression is also colored by their other responses, not simply this one in isolation. Taking them together, the impression is that while they're willing to take money and start allowing you rely on them for archived data... they aren't going to guarantee they'll be around unless they make enough money to make it worth it for them to even both. Again, honesty I really do appreciate, though I can't agree with the approach. You may say that's what everyone would do, but I disagree, I think great products come from a focus on customer value and a willingness to do the hard-work to provide that value. I believe my statement communicates that far more than theirs.
- xfer 9y ago
  https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio - python library. Tooling is not the best, which is precisely why anybody would bother paying for their service. You can do lots of fancy stuff with warc and cdx index. Check out WAIL(https://github.com/N0taN3rd/wail https://github.com/N0taN3rd/wail), it works great, but it's slow and resource heavy. EDIT: You might also want to try out webrecorder self-hosted. Both use the same library, pywb for archiving.
leepowers 9y ago
Do you have an API? Even if WARC isn't something you feel comfortable tackling an API or webhooks would enable other enterprising developers to add this feature. Some commenters are knocking PageDash for being a cloud service, and that's understandable. You'll have an easier time on-boarding HN users if you make it easy for them to access and control their data.
- ernsheong 9y ago
  This is something some folks have been asking. I think it's still early days, but it's something for a bit later.

bravura 9y ago

I'm not sure why this is getting downvoted. WARC tooling is pretty terrible. I hack data and text for a living and working with WARC was extremely difficult compared to the simplicity of what I was trying to do. Something that would have taken me five minutes (extract body text) took me about 60 - 90 minutes because the first few libraries I tried didn't work easily.

toomuchtodo 9y ago
Both wget and it’s python spiritual successor wpull support WARC creation natively.
gooseus 9y ago
I don't think it's getting downvoted because WARC is actually easy, I think it's because the creator says they can't be bothered to figure out the web archiving standard before asking people to start giving them money for their web archiving product. The fact that WARC tooling is terrible is the opportunity this person should be leveraging to create value, imo.
- j_s 9y ago
  I will reference this thread when I need an example of minimum viable product catching flak. Putting off WARC makes financial sense.
  - gooseus 9y ago
    The response wasn't: "I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. If it's an essential feature to make this product valuable then I will definitely make it a priority." If it was "I felt overwhelmed...", "...maybe..." and "...no promises". I appreciate the honesty, but it doesn't inspire confidence and I think the flak is more a reaction to the response regarding WARC than the lack of WARC itself.
    - j_s 9y ago
      True. My take-away: don't overshare motivations for decisions in response to MVP feedback. Instead: respond to criticism with a minimal deflection stating "it's on the roadmap".
    - jasonlotito 9y ago
      I feel both your suggested response and the given response amount to the same thing. I mean, looking at what you said and what they said side by side, and how I read it: > Admittedly I bypassed WARC completely as I felt overwhelmed by its technicalities in favor of how I knew the web worked. > I was planing to support WARC, but felt that it was far too complicated to put off the launch of this MVP. Both mention looking at using WARC, and both mention it being complicated. Both say it's not yet available. > If I have a better understanding of WARC maybe that can be done, but I make no promises. > If it's an essential feature to make this product valuable then I will definitely make it a priority. Both offer up the potential for supporting WARC. Neither promise its delivery. The former provides a bit more justification as to the condition for its inclusion (technical capabilities), while the later hides this fact. > I appreciate the honesty, Honesty was the only difference between the response you hoped for and the response provided.
      - gooseus 9y ago
        Really?? The tone and focus of the two statements are entirely different and communicate completely different impressions of the creators relationship to the question/criticism. I mean I don't really want to get pedantic here, but... I will anyways, weird mood I suppose: My statement implies an understanding of the value of WARC and an intention to implement the feature, but tempered by a desire to understand the needs of the customer before taking on the task. Their response indicates a lack of understanding of the technology (some of the honesty that I appreciate, fyi) and offers a non-commitment to implementing it, even IF they were able to understand it. I'll admit my impression is also colored by their other responses, not simply this one in isolation. Taking them together, the impression is that while they're willing to take money and start allowing you rely on them for archived data... they aren't going to guarantee they'll be around unless they make enough money to make it worth it for them to even both. Again, honesty I really do appreciate, though I can't agree with the approach. You may say that's what everyone would do, but I disagree, I think great products come from a focus on customer value and a willingness to do the hard-work to provide that value. I believe my statement communicates that far more than theirs.
xfer 9y ago
https://github.com/webrecorder/warcio https://github.com/webrecorder/warcio - python library. Tooling is not the best, which is precisely why anybody would bother paying for their service. You can do lots of fancy stuff with warc and cdx index. Check out WAIL(https://github.com/N0taN3rd/wail https://github.com/N0taN3rd/wail), it works great, but it's slow and resource heavy. EDIT: You might also want to try out webrecorder self-hosted. Both use the same library, pywb for archiving.

leepowers 9y ago

Do you have an API? Even if WARC isn't something you feel comfortable tackling an API or webhooks would enable other enterprising developers to add this feature. Some commenters are knocking PageDash for being a cloud service, and that's understandable. You'll have an easier time on-boarding HN users if you make it easy for them to access and control their data.

ernsheong 9y ago
This is something some folks have been asking. I think it's still early days, but it's something for a bit later.