6 ms·

> “Currently, nobody uses time except for maybe Spanner at Google, CockroachDB or someone doing database things,” Rosenblum said. “We believe that there’s a lot

> “Currently, nobody uses time except for maybe Spanner at Google, CockroachDB or someone doing database things,” Rosenblum said. “We believe that there’s a lot more places, especially as more and more time-critical things came up. We can do time sync, since we figured out how to do that pretty well. And so we asked: is this part of a trend where we’re going to start programming these systems differently? And [researchers] got kind of excited about that possibility of us being able to pull this off.” Who gave this guy who's apparently never heard of kerberos, ceph, SAML, or any other technology the rights to interview? Sure, their windows are larger, but "nobody uses time" is so clueless it blows the mind, and it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea

nikanj 5y ago
To be fair, their pitch deck did say both Machine Learning and Stanford
deleted 5y ago
[deleted]
galeaspablo 5y ago
Yup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT. The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. I’d love to see some papers, I went through all of Balaji Prabhakar‘s publications (titles only), and didn’t see a single paper that sounded like “NTP but machine learning. If anyone else knows more about this, I’d love to hear from you. Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?
- kloading 5y ago
  I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article: Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at explaining the general idea below. Network edge-based measurement (one-way delay?): https://www.usenix.org/system/files/nsdi19-geng.pdf https://www.usenix.org/system/files/nsdi19-geng.pdf Congestion Control: https://www.usenix.org/system/files/nsdi21-liu.pdf https://www.usenix.org/system/files/nsdi21-liu.pdf Each paper I've listed builds on the last. Their method to synchronize clocks made accurate and efficient measurement of one-way delay possible with commodity hardware. This measurement of one-way delay occurs at the edge, allowing them to "hold" incoming packets at the edge for extremely small periods to reduce congestion (latency) while maintaining throughput. From my understanding, traditional congestion control algorithms require rich telemetry from the entire network, which is likely not accessible in a public cloud environment. Balaji and clockwork's algorithms only need to make these measurements from the edge (which customers in public cloud have access to). I'm curious to see how all this will scale for multi-region deployments. If the latency between VMs from region 1 and region 2 is significant, I wonder if the measurement will actually be useful in deciding to "hold" the packets.
  - galeaspablo 5y ago
    This is exactly what I was looking for. Thank you. It’s a shame fundamental publications aren’t part of PR articles. I’d also love to see better details on their website. Oh well, I cant expect everything to work like academia — c’est la vie.
  - shiftingleft 5y ago
    > I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. The first article you reference includes something about Support Vector Machines (SVMs).
  - danzheng 5y ago
    I'm from the Clockwork team, thanks for listing the relevant papers. Accurate clocks sync enables true one-way delay measurements (instead of RTT/2), this allows for edge-based network visibility. We launched Latency Sensei beta – a sensor, monitor and auditor that provides visibility into cloud deployments. The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports include: 1)how VM colocation impairs network bandwidth, and 2) tale of 2 cloud regions, London vs Singapore. Take a look and we'd love to get some feedback https://sensei.clockwork.io/user/gallery/ https://sensei.clockwork.io/user/gallery/ On Congestion control, an edge-based solution is coming soon. If you're interested in a private beta, email us at hello@clockwork.io
- isatty 5y ago
  > Surely $21M doesn’t get dropped with at least someone doing due dill on the tech? You’d be surprised, I was.
- insaneirish 5y ago
  > The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. They have the 'names' because they are a reputable group of people. They have been at this problem for a while, first with extensive research that required clock sync as a pre-requisite, and then to evolving into clock sync as a formidable problem unto itself. Clockwork is a rename of the company as far as I can tell. Their original name (Tick Tock Networks) [1] was probably too close to what has become a very popular homophone. [1]: https://opencorporates.com/companies/us_de/6947372 https://opencorporates.com/companies/us_de/6947372
  - galeaspablo 5y ago
    Sure, but ‘names’ don’t explain how they do what they claim to do. Another reply kindly provided references to publications (which I couldn’t find myself) — they made for great reading. You seem to know their work, so if you have further publications, I’d love to get them please.
    - insaneirish 5y ago
      Here's a talk Balaji Prabhakar gave: https://www.youtube.com/watch?v=YQSA99nrFEU https://www.youtube.com/watch?v=YQSA99nrFEU. Looks like the clock sync part starts about 30 minutes in. Haven't watched this one, but might be complementary: https://www.youtube.com/watch?v=Opf9CBwP5R8 https://www.youtube.com/watch?v=Opf9CBwP5R8.
- unmole 5y ago
  > Surely $21M doesn’t get dropped with at least someone doing due dill on the tech? Juicero raised $120M.
  - charcircuit 5y ago
    Juicero only raised $18.5M in their series A.
    - unmole 5y ago
      And?
- ahazred8ta 5y ago
  From the article, time synchronization is only a small part of what they do. Their Big Thing is traffic shaping and latency management within and between datacenters.
  - galeaspablo 5y ago
    Those products are built on time synchronization. E.g., a latency measurement needs a _true_ difference between the timestamps of two servers.
- jacquesm 5y ago
  > Surely $21M doesn’t get dropped with at least someone doing due dill on the tech? Just some words: Theranos. Ubeam. Moller. And many, many others. Investors do this all the time. FOMO + 'wouldn't it be nice' are powerful ways to part fools from their money.
paxys 5y ago
None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem. So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense. If these folks can achieve what they want at a couple orders of magnitude cheaper than current prices then you are absolutely going to see a lot more regular use cases show up to take advantage of it.
- Gelob 5y ago
  Telco workloads and 5g require precise time using technologies like ptp. The cheaper, more commodity those solutions get the better.
  - _3u10 5y ago
    If only these devices had a clock capable of measuring time to within 1 nanosecond. If one has such an accurate clock they might be able to accurately plot their position on earth to within 1 ft merely by checking the timed pulses of orbiting satellites.
    - jazzyjackson 5y ago
      I guess you're downvoted for being sarcastic or maybe people don't know 20 dollar GPS chips provide a ~10 nanosecond resolution clock
      - SAI_Peregrinus 5y ago
        And GPS-disciplined oven controlled crystal oscillators (OCXOs) provide some of the best frequency accuracy and stability around. They range from about a hundred dollars up through a few thousand, depending on phase noise.
      - _3u10 5y ago
        Haha I have no idea. But yeah just call the GPS algo a Time AI and voila, $21 m in funding to go get some kids in shenzen to put a GPS chip on a USB, PCIe, NVME slot. Hey look we got a GPU to learn the Kalman filter algo. Many are actually better than 10 ns basically a 1 foot CEP is 1 ns. If they are fixed and always on the accuracy it can build over a day is incredible.
chillfox 5y ago
Reminds me of early in my career where I encountered a home made time sync system because the creator had never heard of NTP.
- jiveturkey 5y ago
  How so? Clearly these guys have heard of NTP. They are talking about ns precision systems, not ms.
- stevekemp 5y ago
  20 years ago I went on a ridealong with a bunch of men who were changing the times on the clocks installed in a bunch of church towers. It was fascinating to get to watch the process. I'd always taken it for granted the big hotels, railway stations, and churches would often have big clocks you could see from a distance and that they'd be self-correcting. But of course a lot of old clocks are purely mechanical so when the time changed an hour forwards/backwards somebody would need to physically change them.
  - pmontra 5y ago
    Even the clock in my car (2016) doesn't self correct. It goes ahead of about 1 minute every month.
jrowley 5y ago
Exactly! what would Leslie lamport say? ;) https://lamport.azurewebsites.net/pubs/time-clocks.pdf https://lamport.azurewebsites.net/pubs/time-clocks.pdf
oefrha 5y ago
[flagged]
- Nextgrid 5y ago
  Given how much complete garbage has been VC-funded (or rather, overfunded, since the correct amount of funding should've been $0) it's good to remain skeptical.
  - oefrha 5y ago
    If your way of remaining skeptical is opening the HN discussion and jumping into the first crappy hot take concluding that all people involved are clueless idiots without even skimming the article, your skepticism doesn’t mean much.
- evol262 5y ago
  Example of a garbage HN drive-by comment: this. I've spent a _long time_ working in distributed systems with components which are "close to the metal". There are use cases for this, many of which are better solved by "hardware timestamping NICs" and "run a stratum 1/2 NTP server with high locality (and if you're geo-distributed, run multiple on systems dedicated to that purpose, because ensuring you're within nanoseconds of _someone else's_ infrastructure is generally an order or magnitude less important than internal coherence). That was the best possible quote from the author/subject trying to explain the use cases. Did you even read TFA? It's completely unclear from their article what the intended uses cases are, what the problem space is, and how much working knowledge they have of existing solutions. Naming people who invested in other companies who are investing in this one and showing what looks like a dashboard for a timekeeping solution is normal TechCrunch garbage, but that quote was a step above. Comments like yours (going all the way back to /.) days really just tell me that you didn't read the article, and it makes it look like your primary goal is sophistry.
unfocussed_mike 5y ago
> it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea Bless you. Are you younger than 40? :-) Tech people who are over about 40 have, alas, no trouble imagining this at all!

nikanj 5y ago

To be fair, their pitch deck did say both Machine Learning and Stanford

deleted 5y ago

[deleted]

galeaspablo 5y ago

Yup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT. The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. I’d love to see some papers, I went through all of Balaji Prabhakar‘s publications (titles only), and didn’t see a single paper that sounded like “NTP but machine learning. If anyone else knows more about this, I’d love to hear from you. Surely $21M doesn’t get dropped with at least someone doing due dill on the tech?

kloading 5y ago
I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article: Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at explaining the general idea below. Network edge-based measurement (one-way delay?): https://www.usenix.org/system/files/nsdi19-geng.pdf https://www.usenix.org/system/files/nsdi19-geng.pdf Congestion Control: https://www.usenix.org/system/files/nsdi21-liu.pdf https://www.usenix.org/system/files/nsdi21-liu.pdf Each paper I've listed builds on the last. Their method to synchronize clocks made accurate and efficient measurement of one-way delay possible with commodity hardware. This measurement of one-way delay occurs at the edge, allowing them to "hold" incoming packets at the edge for extremely small periods to reduce congestion (latency) while maintaining throughput. From my understanding, traditional congestion control algorithms require rich telemetry from the entire network, which is likely not accessible in a public cloud environment. Balaji and clockwork's algorithms only need to make these measurements from the edge (which customers in public cloud have access to). I'm curious to see how all this will scale for multi-region deployments. If the latency between VMs from region 1 and region 2 is significant, I wonder if the measurement will actually be useful in deciding to "hold" the packets.
- galeaspablo 5y ago
  This is exactly what I was looking for. Thank you. It’s a shame fundamental publications aren’t part of PR articles. I’d also love to see better details on their website. Oh well, I cant expect everything to work like academia — c’est la vie.
- shiftingleft 5y ago
  > I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. The first article you reference includes something about Support Vector Machines (SVMs).
- danzheng 5y ago
  I'm from the Clockwork team, thanks for listing the relevant papers. Accurate clocks sync enables true one-way delay measurements (instead of RTT/2), this allows for edge-based network visibility. We launched Latency Sensei beta – a sensor, monitor and auditor that provides visibility into cloud deployments. The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports include: 1)how VM colocation impairs network bandwidth, and 2) tale of 2 cloud regions, London vs Singapore. Take a look and we'd love to get some feedback https://sensei.clockwork.io/user/gallery/ https://sensei.clockwork.io/user/gallery/ On Congestion control, an edge-based solution is coming soon. If you're interested in a private beta, email us at hello@clockwork.io
isatty 5y ago
> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech? You’d be surprised, I was.
insaneirish 5y ago
> The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. They have the 'names' because they are a reputable group of people. They have been at this problem for a while, first with extensive research that required clock sync as a pre-requisite, and then to evolving into clock sync as a formidable problem unto itself. Clockwork is a rename of the company as far as I can tell. Their original name (Tick Tock Networks) [1] was probably too close to what has become a very popular homophone. [1]: https://opencorporates.com/companies/us_de/6947372 https://opencorporates.com/companies/us_de/6947372
- galeaspablo 5y ago
  Sure, but ‘names’ don’t explain how they do what they claim to do. Another reply kindly provided references to publications (which I couldn’t find myself) — they made for great reading. You seem to know their work, so if you have further publications, I’d love to get them please.
  - insaneirish 5y ago
    Here's a talk Balaji Prabhakar gave: https://www.youtube.com/watch?v=YQSA99nrFEU https://www.youtube.com/watch?v=YQSA99nrFEU. Looks like the clock sync part starts about 30 minutes in. Haven't watched this one, but might be complementary: https://www.youtube.com/watch?v=Opf9CBwP5R8 https://www.youtube.com/watch?v=Opf9CBwP5R8.
unmole 5y ago
> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech? Juicero raised $120M.
- charcircuit 5y ago
  Juicero only raised $18.5M in their series A.
  - unmole 5y ago
    And?
ahazred8ta 5y ago
From the article, time synchronization is only a small part of what they do. Their Big Thing is traffic shaping and latency management within and between datacenters.
- galeaspablo 5y ago
  Those products are built on time synchronization. E.g., a latency measurement needs a _true_ difference between the timestamps of two servers.
jacquesm 5y ago
> Surely $21M doesn’t get dropped with at least someone doing due dill on the tech? Just some words: Theranos. Ubeam. Moller. And many, many others. Investors do this all the time. FOMO + 'wouldn't it be nice' are powerful ways to part fools from their money.

paxys 5y ago

None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem. So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense. If these folks can achieve what they want at a couple orders of magnitude cheaper than current prices then you are absolutely going to see a lot more regular use cases show up to take advantage of it.

Gelob 5y ago
Telco workloads and 5g require precise time using technologies like ptp. The cheaper, more commodity those solutions get the better.
- _3u10 5y ago
  If only these devices had a clock capable of measuring time to within 1 nanosecond. If one has such an accurate clock they might be able to accurately plot their position on earth to within 1 ft merely by checking the timed pulses of orbiting satellites.
  - jazzyjackson 5y ago
    I guess you're downvoted for being sarcastic or maybe people don't know 20 dollar GPS chips provide a ~10 nanosecond resolution clock
    - SAI_Peregrinus 5y ago
      And GPS-disciplined oven controlled crystal oscillators (OCXOs) provide some of the best frequency accuracy and stability around. They range from about a hundred dollars up through a few thousand, depending on phase noise.
    - _3u10 5y ago
      Haha I have no idea. But yeah just call the GPS algo a Time AI and voila, $21 m in funding to go get some kids in shenzen to put a GPS chip on a USB, PCIe, NVME slot. Hey look we got a GPU to learn the Kalman filter algo. Many are actually better than 10 ns basically a 1 foot CEP is 1 ns. If they are fixed and always on the accuracy it can build over a day is incredible.

chillfox 5y ago

Reminds me of early in my career where I encountered a home made time sync system because the creator had never heard of NTP.

jiveturkey 5y ago
How so? Clearly these guys have heard of NTP. They are talking about ns precision systems, not ms.
stevekemp 5y ago
20 years ago I went on a ridealong with a bunch of men who were changing the times on the clocks installed in a bunch of church towers. It was fascinating to get to watch the process. I'd always taken it for granted the big hotels, railway stations, and churches would often have big clocks you could see from a distance and that they'd be self-correcting. But of course a lot of old clocks are purely mechanical so when the time changed an hour forwards/backwards somebody would need to physically change them.
- pmontra 5y ago
  Even the clock in my car (2016) doesn't self correct. It goes ahead of about 1 minute every month.

jrowley 5y ago

Exactly! what would Leslie lamport say? ;) https://lamport.azurewebsites.net/pubs/time-clocks.pdf https://lamport.azurewebsites.net/pubs/time-clocks.pdf

oefrha 5y ago

[flagged]

Nextgrid 5y ago
Given how much complete garbage has been VC-funded (or rather, overfunded, since the correct amount of funding should've been $0) it's good to remain skeptical.
- oefrha 5y ago
  If your way of remaining skeptical is opening the HN discussion and jumping into the first crappy hot take concluding that all people involved are clueless idiots without even skimming the article, your skepticism doesn’t mean much.
evol262 5y ago
Example of a garbage HN drive-by comment: this. I've spent a _long time_ working in distributed systems with components which are "close to the metal". There are use cases for this, many of which are better solved by "hardware timestamping NICs" and "run a stratum 1/2 NTP server with high locality (and if you're geo-distributed, run multiple on systems dedicated to that purpose, because ensuring you're within nanoseconds of _someone else's_ infrastructure is generally an order or magnitude less important than internal coherence). That was the best possible quote from the author/subject trying to explain the use cases. Did you even read TFA? It's completely unclear from their article what the intended uses cases are, what the problem space is, and how much working knowledge they have of existing solutions. Naming people who invested in other companies who are investing in this one and showing what looks like a dashboard for a timekeeping solution is normal TechCrunch garbage, but that quote was a step above. Comments like yours (going all the way back to /.) days really just tell me that you didn't read the article, and it makes it look like your primary goal is sophistry.

unfocussed_mike 5y ago

> it's hard to imagine who decided that "NTP, but machine learning" is a $21M idea Bless you. Are you younger than 40? :-) Tech people who are over about 40 have, alas, no trouble imagining this at all!