4 ms·

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are ther

by tabeth 9y ago

I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. Are there any articles that supplement this that explain how much business value is added/lost by the existence/removal of these kind of features? In the case of NYT I suspect its popularity is maintained because of the perception (real or not) of high quality journalism, in spite of any technical failings. --- How much would be lost if NYT was just implemented as text articles that are cached and styled with some CSS. "Personalization" could be added by tags each article has and a small component that shows the three most recent articles that share the same tag.

weego 9y ago
I can't speak for the situation at the NYT but the actual public site for online papers are often pretty simple things with most of the complexity being ad logic. The systems here almost entirely deal with writing and content retrieval pipelines for stuff that was written years ago in other systems that isn't tagged or stored in sympathetic ways, and there will also be the very old school print pipeline to have to deal with too.
- manigandham 9y ago
  How much time/effort does it take to just update all this old stuff once and for all?
  - dangayle 9y ago
    Literally teams of interns manually re-typing old articles from microfilm. OCR isn't quite there yet, not for dealing with the ways newspapers and newspaper design has changed over the years. You'd get the text, but there would be no guarantees that it that story bylines, headlines, factboxes, and photos match. Our own digital archives go back to 1994, anything before that is manual input.
cookiecaper 9y ago
>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is understated because these experimental setups misinterpret the functionality of the new architectures/formats they're using. It doesn't truly rear its ugly head until there is a major data loss or corruption event. It's not that these never happen with RDBMS, it's just that RDBMS contemplates this possibility and tries to make it pretty hard to do that, whereas message queues just automatically delete stuff (by design, so they can serve as functional message queues!). RDBMS have spoiled us and we take its featureset, 40+ years in the making, for granted. We need to be careful and not assume that `GROUP BY` is the only thing we leave on the table when we "adopt" (more accurately abuse) one of these new-wave solutions as a system of record. Since no one is going to admit to their boss "this wouldn't have happened if we used Postgres", and since most bosses are not going to know what that means, most of these spectacular failures will never be accurately attributed to their true cause: developers putting their interest in trying new things above their duty to ensure their employer's systems are reliable, stable, and resilient.
- awinder 9y ago
  There are non-negligible problems in the news space like: 1. Supporting full-text search for a fair number of concurrent users 2. Availability of the system with minimal downtime 3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic 4. Notifications I could go on and on but honestly, it's just a tone-deaf response. Parting pot-shot: "No one is going to admit to their boss that the reason a worldwide news organization can't publish any stories is because their one postgres master node went down, or is waiting on a state transfer to a fallback master"
  - cookiecaper 9y ago
    "We can't publish right now because the database had to enter an unplanned maintenance period" is a lot different from "our authoritative archive is gone and we have to try to rebuild it from all these separate 'materialized views', woops."
feelin_googley 9y ago
"I wonder how much this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied." Do you mean only the developers (==very smart people) hired by the NYT or those hired at other companies, too?
1wheel 9y ago
> How much would be lost if NYT was just implemented as text articles that are cached and styled with some CSS. https://www.nytimes.com/interactive/2016/12/28/us/year-in-interactive-graphics.html https://www.nytimes.com/interactive/2016/12/28/us/year-in-in...
- tabeth 9y ago
  That's neat, but most of those aren't anything that can't be implemented with just CSS and HTML, unlike some of the stuff you've done for the NYT. Though even in that case, isn't all of this stuff just static assets? Hasn't this problem already been "solved"?

by tabeth 9y ago

weego 9y ago

I can't speak for the situation at the NYT but the actual public site for online papers are often pretty simple things with most of the complexity being ad logic. The systems here almost entirely deal with writing and content retrieval pipelines for stuff that was written years ago in other systems that isn't tagged or stored in sympathetic ways, and there will also be the very old school print pipeline to have to deal with too.

manigandham 9y ago
How much time/effort does it take to just update all this old stuff once and for all?
- dangayle 9y ago
  Literally teams of interns manually re-typing old articles from microfilm. OCR isn't quite there yet, not for dealing with the ways newspapers and newspaper design has changed over the years. You'd get the text, but there would be no guarantees that it that story bylines, headlines, factboxes, and photos match. Our own digital archives go back to 1994, anything before that is manual input.

cookiecaper 9y ago

>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied. That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome. The raw cost is understated because these experimental setups misinterpret the functionality of the new architectures/formats they're using. It doesn't truly rear its ugly head until there is a major data loss or corruption event. It's not that these never happen with RDBMS, it's just that RDBMS contemplates this possibility and tries to make it pretty hard to do that, whereas message queues just automatically delete stuff (by design, so they can serve as functional message queues!). RDBMS have spoiled us and we take its featureset, 40+ years in the making, for granted. We need to be careful and not assume that `GROUP BY` is the only thing we leave on the table when we "adopt" (more accurately abuse) one of these new-wave solutions as a system of record. Since no one is going to admit to their boss "this wouldn't have happened if we used Postgres", and since most bosses are not going to know what that means, most of these spectacular failures will never be accurately attributed to their true cause: developers putting their interest in trying new things above their duty to ensure their employer's systems are reliable, stable, and resilient.

awinder 9y ago
There are non-negligible problems in the news space like: 1. Supporting full-text search for a fair number of concurrent users 2. Availability of the system with minimal downtime 3. Scalability within the day and year, traffic patterns around e.g., breaking news events will far surpass 2AM traffic 4. Notifications I could go on and on but honestly, it's just a tone-deaf response. Parting pot-shot: "No one is going to admit to their boss that the reason a worldwide news organization can't publish any stories is because their one postgres master node went down, or is waiting on a state transfer to a fallback master"
- cookiecaper 9y ago
  "We can't publish right now because the database had to enter an unplanned maintenance period" is a lot different from "our authoritative archive is gone and we have to try to rebuild it from all these separate 'materialized views', woops."

feelin_googley 9y ago

"I wonder how much this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied." Do you mean only the developers (==very smart people) hired by the NYT or those hired at other companies, too?

1wheel 9y ago

> How much would be lost if NYT was just implemented as text articles that are cached and styled with some CSS. https://www.nytimes.com/interactive/2016/12/28/us/year-in-interactive-graphics.html https://www.nytimes.com/interactive/2016/12/28/us/year-in-in...

tabeth 9y ago
That's neat, but most of those aren't anything that can't be implemented with just CSS and HTML, unlike some of the stuff you've done for the NYT. Though even in that case, isn't all of this stuff just static assets? Hasn't this problem already been "solved"?