16 ms·

From the notice: > It affected the system that generates up-to-date schedules for trains and staff. ...boy oh boy did trying to look into what NS uses for cre

by quartz 5y ago

From the notice: > It affected the system that generates up-to-date schedules for trains and staff. ...boy oh boy did trying to look into what NS uses for crew scheduling ever send me down a rabbit hole. I don't know if the systems have changed but while poking around online I found this[1] doc about the Netherlands' timetable revamp around 2006 and it talks about the complexity of TURNI-- their on-the-fly crew scheduling system. > A typical workday at NS includes approximately 15,000 trips for drivers and 18,000 for conductors. The resulting number of duties is approximately 1,000 for drivers and 1,300 for conductors. This leads to extremely difficult crew scheduling instances. Nevertheless, because of the highly sophisticated applied algorithms, TURNI solves these cases in 24hours of computing time on a personal computer. Therefore, we can construct all crew schedules for all days of the week within just a few days. Then I found more detail about TURNI's implementation in this[2] paper about optimizing crew scheduling for timetables. > In the railway industry the sizes of the crew scheduling instances are, in general, a magnitude larger than in the airline industry. Moreover, crew can be relieved during the drive of a train resulting in much more trips per duty than typical in airlines. In other words, the combinatorial explosion is much higher. The latter has made the application of these models in the railway industry prohibitive until recently. Cool stuff. Finally, gleaning from ns.nl's careers page[3] everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them. [1] https://homepages.cwi.nl/~lex/files/Interfaces.pdf https://homepages.cwi.nl/~lex/files/Interfaces.pdf [2] https://repub.eur.nl/pub/11701/ei200803.pdf https://repub.eur.nl/pub/11701/ei200803.pdf [3] https://werkenbijns.nl/werkgebieden/it/sap-specialist-bij-ns/ https://werkenbijns.nl/werkgebieden/it/sap-specialist-bij-ns... sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.

deleted 5y ago
[deleted]
incompatible 5y ago
If they are scheduling days in advance, then why would a failure of the scheduling system cause an instant outage?
- jonny_eh 5y ago
  Maybe a mistake was detected and they needed to recalculate?
- tyingq 5y ago
  It said for "schedules for trains and staff". Crew scheduling is complicated and changes constantly.
- naikrovek 5y ago
  because accidents and delays happen anytime, and if the schedule can't adapt, you can't run the trains. people seem to believe that running trains is some simple thing. it is the exact opposite of a simple thing, even for small systems with only a few dozen trains
- anticensor 5y ago
  Why not fall back to old-school time and distance interval running?
xyst 5y ago
I have seen “SAP” tossed around in this industry and still have no clue what they do to earn money. Are they the equivalence of “salesforce”?
- liamwire 5y ago
  In a word: ERP.
- lmpdev 5y ago
  SAP are more known for ERP than CRM They're absolutely colossal in size My ex-girlfiend's father is an engineer there in Baden-Wurttemberg - I'd characterise them as closer to something like Oracle rather than Salesforce
  - nine_k 5y ago
    They are so colossal that switching to SAP usually means reworking company's processes to match SAP's, not the other way around.
    - shdon 5y ago
      This was true even for Royal Dutch Shell, one of the largest companies in the world. A few years before he retired, my dad led a team responsible for coordinating implementation of SAP and for training thousands or even tens of thousands of employees to work with the new system.
    - GekkePrutser 5y ago
      To be fair, that is the case for most ERPs. It's not really about the size of the product but rather the important role it plays in a company. Most ERPs can be quite flexible too, there's a ton of customisation possible. And a whole cottage industry of addons and integrations for every imaginable integration or business model. But the problem is you really screw yourself with added complexity when it's time to update. Then you're really condemning yourself to spending millions on consultants at a time when you might not have the resources. And postponement leads to technical debt. Hence the idea of adapting the business to the system. Making sense too if you think about it this way: you're not really just buying a system but a methodology. And another thing is that it makes integration with other companies easier. Makes the company more easy to sell or too acquire other companies and integrate them.
samstave 5y ago
I love deep diving on public info to determine the internals (even, not only SOPs, policies, but also *governing culture* can be easily gleaned from reading what is placed in public view. For example, The Salvation Army is one of the biggest jokes in all of charitable history. I wont go into details here, but I would NEVER give a dime to the Salvation "army" (of free slave labor under the guise of "salvation" (its a tax and money fraud to its core (I went through a SHIT ton of financial documents to see what their true operatin is disguised as. AVOID) TL;DR: TSA uses its place as a church-like service to get special tax incentives, even the buildings in SF were donated to them. They run a for-profit-grey-market of boutiques they supply with "donations" they have running trades with China-Town in SF. They take subsidies, donations and other benefits. They require all that they "help" sign up for state benefits, sign the benefits over to their org, take the benefits, then feed donated & expired processed fod to their labor force. They only take in able bodied. Force them to go to church each sunday. and make them use external addiction help, such as AA and make it clear they are not a treatment organiztion. They maintain a very small amount of actual employees, and play on the egos of those in their "program" using military style rankings to appease the egos of the ignorant... They then profit like mad, corruption is rampant and they feed all their "good" donatins to their side hustle of boutiques and other grey market. Its a fn racket. =--- The point being that if you DD a company/org, you can often find out a lot from financial public info, comments from even just a few employees etc..
- nicolas_t 5y ago
  Please go into details, I always think it's important to call out charities that are fraudulent and going into details helps with detecting if other charities are fraudulent or not. What lead you to look into this? How did you originally suspect them of being fraudulent? What clues did you see in their financial documents? For example, I tend to look at Charity Navigator and for Salvation Army, they mostly get dinged by not having an independent board (which to be fair is a clear red flag) but not much more, so as someone who does want any contribution I make to charities to go toward their cause, I'm very interested in any methodology around researching a charity.
- wolverine876 5y ago
  Do you have any evidence of these claims? A news article, for example? I knew someone who worked briefly at the Salvation Army. They said it was shockingly efficient and effective at its charitable mission.
MetaWhirledPeas 5y ago
> everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them. I have a reflex repulsion for off-the-shelf "enterprise" software. You pay millions of dollars to subscribe to a suite of (seemingly) bad, bloated software that takes more work to customize and integrate than it would to fully replace. An army of developers trying to keep a business running with the world's largest Swiss Army knife. My gut tells me that big names like SAP and Sales Force are easier to sell to executives, and it makes a company seem like it knows what it's doing. And maybe feeling pressure to choose a "respected" brand leads to fewer alternatives? Or maybe all the alternatives are quickly bought out by the behemoths. I really don't know. Maybe someone can enlighten me. It just seems like an enormous turd of inefficiency to me.
- resonious 5y ago
  My previous boss recently tried to pick up an enterprise solution. He runs a non-software company that has a lot of custom internal apps - 90% built/maintained by interns. Of course the intern-built custom apps are super buggy but the bugs get fixed immediately, and the system is tailored to the company's way of doing things. The company spent an obscene amount of money just demoing a full-fledged SAP-type thing to potentially replace the custom stuff, but they ultimately found the rag-tag web apps better and cancelled the SAP contract. That whole situation convinced me pretty thoroughly that "enterprise" software is indeed a form of large-scale snake oil. I think most businesses can go a long way with just spreadsheets and email.
  - sebmellen 5y ago
    This is one of the reasons I'm bullish on no-code doc-apps like Coda [0]. I recently started consulting with someone who calls himself "The Coda Guy" and pulls together truly wizard-like apps out of Coda [1]. (Coda is effectively a hybrid of Excel, Google Docs, and Notion.) I think the "Excel+" category should receive a lot more attention than it does now. I can easily see "no-code" tools like Coda taking over Salesforce and effectively filling the niche you describe. [0]: https://coda.io https://coda.io [1]: https://coda.io/@simpladocs/cycling-specific-template https://coda.io/@simpladocs/cycling-specific-template / https://coda.io/@simpladocs/crm-interaction-tracker https://coda.io/@simpladocs/crm-interaction-tracker
rgj 5y ago
> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel. Those are pretty standard terms here in .NL. 36 hours is considered fulltime by most employers, including government and semi-government. Pension is offered almost everywhere and at least 4 weeks vacation is the legal minimum. Maybe except for the _unlimited_ train travel but most employers do offer free train travel between home and work. This also means that if an IT system goes down on a Sunday, a lot of employees won’t even pick up the phone until Monday 9 AM.
- t0mas88 5y ago
  Indeed pretty standard, even in non-government it would be 40 hours but all other terms the same and most likely a company car + private use of it instead of the train. And I'm also sure they have a significant on-call setup of the IT teams for any out of office hours issues, just like every other large 24x7 kind of organization.
  - rgj 5y ago
    Of course they do. But the real specialists will most likely have been able to keep themselves out of the on call schedule.
- ChuckNorris89 5y ago
  >36 hours is considered fulltime by most employers How standard is that in the private sector or non former state owned enterprises? I haven't looked at the NL market in a while but most tech jobs I saw back then were 40h/week.
  - AndyMcConachie 5y ago
    Most private sector tech jobs in NL are 40 hours per week. But many private sector jobs are also flexible in ways that I've never seen in the USA. For example, allowing an employee to work 40 hours in 4 days instead of 5.
weq 5y ago
I worked in defence on systems that did similiar and i remmember the one my company inherited and wow and wow it was the most complex, nested peice of SQL stored procs i have ever witnessed. It did the job, but it took HOURS to run. We had to re-write it completely to make any changes. But thats what happens when you let companies like Accenture at a system of yours! You pay them the big bux to be sueable, and they milk you like a cow.
- quartz 5y ago
  A lifetime ago I did a stint as an Oracle consultant and 100% this. A gigantic sql query was always the answer no matter the question. I remember using sql queries to generate pdf and ps file output that would stream directly to printer feeds. So many “concat”s in that massive blob and nearly impossible to maintain for the next poor soul who had to own the code. Learned a lot about db best practices to make these systems resilient and reportable but the whole “the database is the system” angle was very much a “every problem is a nail to this hammer” situation.
lultimouomo 5y ago
> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel. And we can assume that a position in the IT department just opened up...
- sofixa 5y ago
  Blameless culture! Firing people for (one-off) mistakes or issues is much more a thing in the US than in most EU countries.
  - gpvos 5y ago
    tbh a few similar outages have happened in recent years, they do seem to have a problem and maybe someone does need to be fired.
sofixa 5y ago
> 36hr week, 5 weeks vacation, pension 1 hour more than the standard French contract, and there are also far more holidays here.. > free unlimited 2nd class + low cost 1st class train travel. .. okay that's actually a great perk. IIRC the French national railway company, SNCF, provide only a limited number of tickets per employee (like 10 per year? someone please correct me if I'm wrong). Their UX is also atrocious though (websites are just shit at workflows, apps are weird and break easily, ticket machines are OK but with really bad touch screens, and train announcement screens are each running Windows XP(with literally hundreds, if not thousands of them on big train stations), while from what i recall NS was acceptable.
- thibautg 5y ago
  I work for the Belgian railways and I have a free unlimited first class pass for Benelux (Belgium, Netherlands and Luxembourg) + lots of international free tickets, also in 1st class. But no company car, obviously. There is no UX: it's just a small sheet of paper.
  - HowTheStoryEnds 5y ago
    The workers don't get that though, they get 2nd class.
tannhaeuser 5y ago
I have zero knowledge of how the Dutch railway scheduling system works, nor of the nature of this failure, but SAP has been blogging recently about using Prolog for combinatorical optimization tasks such as the one that may be in use here or in SAP's Advanced Planning [1]. There are also well-known Prolog proponents from NL. [1]: https://blogs.sap.com/2022/03/11/what-you-may-not-know-of-prolog/ https://blogs.sap.com/2022/03/11/what-you-may-not-know-of-pr...
miggol 5y ago
Thank you for your informative reply! These are such good reads

by quartz 5y ago

deleted 5y ago

[deleted]

incompatible 5y ago

If they are scheduling days in advance, then why would a failure of the scheduling system cause an instant outage?

jonny_eh 5y ago
Maybe a mistake was detected and they needed to recalculate?
tyingq 5y ago
It said for "schedules for trains and staff". Crew scheduling is complicated and changes constantly.
naikrovek 5y ago
because accidents and delays happen anytime, and if the schedule can't adapt, you can't run the trains. people seem to believe that running trains is some simple thing. it is the exact opposite of a simple thing, even for small systems with only a few dozen trains
anticensor 5y ago
Why not fall back to old-school time and distance interval running?

xyst 5y ago

I have seen “SAP” tossed around in this industry and still have no clue what they do to earn money. Are they the equivalence of “salesforce”?

liamwire 5y ago
In a word: ERP.
lmpdev 5y ago
SAP are more known for ERP than CRM They're absolutely colossal in size My ex-girlfiend's father is an engineer there in Baden-Wurttemberg - I'd characterise them as closer to something like Oracle rather than Salesforce
- nine_k 5y ago
  They are so colossal that switching to SAP usually means reworking company's processes to match SAP's, not the other way around.
  - shdon 5y ago
    This was true even for Royal Dutch Shell, one of the largest companies in the world. A few years before he retired, my dad led a team responsible for coordinating implementation of SAP and for training thousands or even tens of thousands of employees to work with the new system.
  - GekkePrutser 5y ago
    To be fair, that is the case for most ERPs. It's not really about the size of the product but rather the important role it plays in a company. Most ERPs can be quite flexible too, there's a ton of customisation possible. And a whole cottage industry of addons and integrations for every imaginable integration or business model. But the problem is you really screw yourself with added complexity when it's time to update. Then you're really condemning yourself to spending millions on consultants at a time when you might not have the resources. And postponement leads to technical debt. Hence the idea of adapting the business to the system. Making sense too if you think about it this way: you're not really just buying a system but a methodology. And another thing is that it makes integration with other companies easier. Makes the company more easy to sell or too acquire other companies and integrate them.

samstave 5y ago

I love deep diving on public info to determine the internals (even, not only SOPs, policies, but also *governing culture* can be easily gleaned from reading what is placed in public view. For example, The Salvation Army is one of the biggest jokes in all of charitable history. I wont go into details here, but I would NEVER give a dime to the Salvation "army" (of free slave labor under the guise of "salvation" (its a tax and money fraud to its core (I went through a SHIT ton of financial documents to see what their true operatin is disguised as. AVOID) TL;DR: TSA uses its place as a church-like service to get special tax incentives, even the buildings in SF were donated to them. They run a for-profit-grey-market of boutiques they supply with "donations" they have running trades with China-Town in SF. They take subsidies, donations and other benefits. They require all that they "help" sign up for state benefits, sign the benefits over to their org, take the benefits, then feed donated & expired processed fod to their labor force. They only take in able bodied. Force them to go to church each sunday. and make them use external addiction help, such as AA and make it clear they are not a treatment organiztion. They maintain a very small amount of actual employees, and play on the egos of those in their "program" using military style rankings to appease the egos of the ignorant... They then profit like mad, corruption is rampant and they feed all their "good" donatins to their side hustle of boutiques and other grey market. Its a fn racket. =--- The point being that if you DD a company/org, you can often find out a lot from financial public info, comments from even just a few employees etc..

nicolas_t 5y ago
Please go into details, I always think it's important to call out charities that are fraudulent and going into details helps with detecting if other charities are fraudulent or not. What lead you to look into this? How did you originally suspect them of being fraudulent? What clues did you see in their financial documents? For example, I tend to look at Charity Navigator and for Salvation Army, they mostly get dinged by not having an independent board (which to be fair is a clear red flag) but not much more, so as someone who does want any contribution I make to charities to go toward their cause, I'm very interested in any methodology around researching a charity.
wolverine876 5y ago
Do you have any evidence of these claims? A news article, for example? I knew someone who worked briefly at the Salvation Army. They said it was shockingly efficient and effective at its charitable mission.

MetaWhirledPeas 5y ago

> everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them. I have a reflex repulsion for off-the-shelf "enterprise" software. You pay millions of dollars to subscribe to a suite of (seemingly) bad, bloated software that takes more work to customize and integrate than it would to fully replace. An army of developers trying to keep a business running with the world's largest Swiss Army knife. My gut tells me that big names like SAP and Sales Force are easier to sell to executives, and it makes a company seem like it knows what it's doing. And maybe feeling pressure to choose a "respected" brand leads to fewer alternatives? Or maybe all the alternatives are quickly bought out by the behemoths. I really don't know. Maybe someone can enlighten me. It just seems like an enormous turd of inefficiency to me.

resonious 5y ago
My previous boss recently tried to pick up an enterprise solution. He runs a non-software company that has a lot of custom internal apps - 90% built/maintained by interns. Of course the intern-built custom apps are super buggy but the bugs get fixed immediately, and the system is tailored to the company's way of doing things. The company spent an obscene amount of money just demoing a full-fledged SAP-type thing to potentially replace the custom stuff, but they ultimately found the rag-tag web apps better and cancelled the SAP contract. That whole situation convinced me pretty thoroughly that "enterprise" software is indeed a form of large-scale snake oil. I think most businesses can go a long way with just spreadsheets and email.
- sebmellen 5y ago
  This is one of the reasons I'm bullish on no-code doc-apps like Coda [0]. I recently started consulting with someone who calls himself "The Coda Guy" and pulls together truly wizard-like apps out of Coda [1]. (Coda is effectively a hybrid of Excel, Google Docs, and Notion.) I think the "Excel+" category should receive a lot more attention than it does now. I can easily see "no-code" tools like Coda taking over Salesforce and effectively filling the niche you describe. [0]: https://coda.io https://coda.io [1]: https://coda.io/@simpladocs/cycling-specific-template https://coda.io/@simpladocs/cycling-specific-template / https://coda.io/@simpladocs/crm-interaction-tracker https://coda.io/@simpladocs/crm-interaction-tracker

rgj 5y ago

> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel. Those are pretty standard terms here in .NL. 36 hours is considered fulltime by most employers, including government and semi-government. Pension is offered almost everywhere and at least 4 weeks vacation is the legal minimum. Maybe except for the _unlimited_ train travel but most employers do offer free train travel between home and work. This also means that if an IT system goes down on a Sunday, a lot of employees won’t even pick up the phone until Monday 9 AM.

t0mas88 5y ago
Indeed pretty standard, even in non-government it would be 40 hours but all other terms the same and most likely a company car + private use of it instead of the train. And I'm also sure they have a significant on-call setup of the IT teams for any out of office hours issues, just like every other large 24x7 kind of organization.
- rgj 5y ago
  Of course they do. But the real specialists will most likely have been able to keep themselves out of the on call schedule.
ChuckNorris89 5y ago
>36 hours is considered fulltime by most employers How standard is that in the private sector or non former state owned enterprises? I haven't looked at the NL market in a while but most tech jobs I saw back then were 40h/week.
- AndyMcConachie 5y ago
  Most private sector tech jobs in NL are 40 hours per week. But many private sector jobs are also flexible in ways that I've never seen in the USA. For example, allowing an employee to work 40 hours in 4 days instead of 5.

weq 5y ago

I worked in defence on systems that did similiar and i remmember the one my company inherited and wow and wow it was the most complex, nested peice of SQL stored procs i have ever witnessed. It did the job, but it took HOURS to run. We had to re-write it completely to make any changes. But thats what happens when you let companies like Accenture at a system of yours! You pay them the big bux to be sueable, and they milk you like a cow.

quartz 5y ago
A lifetime ago I did a stint as an Oracle consultant and 100% this. A gigantic sql query was always the answer no matter the question. I remember using sql queries to generate pdf and ps file output that would stream directly to printer feeds. So many “concat”s in that massive blob and nearly impossible to maintain for the next poor soul who had to own the code. Learned a lot about db best practices to make these systems resilient and reportable but the whole “the database is the system” angle was very much a “every problem is a nail to this hammer” situation.

lultimouomo 5y ago

sofixa 5y ago
Blameless culture! Firing people for (one-off) mistakes or issues is much more a thing in the US than in most EU countries.
- gpvos 5y ago
  tbh a few similar outages have happened in recent years, they do seem to have a problem and maybe someone does need to be fired.

sofixa 5y ago

> 36hr week, 5 weeks vacation, pension 1 hour more than the standard French contract, and there are also far more holidays here.. > free unlimited 2nd class + low cost 1st class train travel. .. okay that's actually a great perk. IIRC the French national railway company, SNCF, provide only a limited number of tickets per employee (like 10 per year? someone please correct me if I'm wrong). Their UX is also atrocious though (websites are just shit at workflows, apps are weird and break easily, ticket machines are OK but with really bad touch screens, and train announcement screens are each running Windows XP(with literally hundreds, if not thousands of them on big train stations), while from what i recall NS was acceptable.

thibautg 5y ago
I work for the Belgian railways and I have a free unlimited first class pass for Benelux (Belgium, Netherlands and Luxembourg) + lots of international free tickets, also in 1st class. But no company car, obviously. There is no UX: it's just a small sheet of paper.
- HowTheStoryEnds 5y ago
  The workers don't get that though, they get 2nd class.

tannhaeuser 5y ago

I have zero knowledge of how the Dutch railway scheduling system works, nor of the nature of this failure, but SAP has been blogging recently about using Prolog for combinatorical optimization tasks such as the one that may be in use here or in SAP's Advanced Planning [1]. There are also well-known Prolog proponents from NL. [1]: https://blogs.sap.com/2022/03/11/what-you-may-not-know-of-prolog/ https://blogs.sap.com/2022/03/11/what-you-may-not-know-of-pr...

miggol 5y ago

Thank you for your informative reply! These are such good reads