4 ms·

ohhhh, we're arguing past eachother. I am unsure how to reconcile. an SMS isn't just "140 characters/bytes" or whatever (i honestly don't care what your defini

ohhhh, we're arguing past eachother. I am unsure how to reconcile. an SMS isn't just "140 characters/bytes" or whatever (i honestly don't care what your definition of "SMS" is). Of course you could fit 140 characters * 1e12 onto 5 hard drives. Where are you going to put the 1PB (for 1e12, but your own cite says it's 1e13, so 10PB) of metadata, minimum? the most barebones amount of metadata you need to actually have actionable "intelligence" is 1KB per message (technically i was able to finagle it to ~1016 bytes.) And that's for every message, even an SMS that is the single character "K". you need the metadata to derive any information from the SMS. "Lunch?" "yeah" "where?" "the place with the wheel" "okay see you in 25, bring Joel" This is what you propose to save. (quick math shows you went off something like ~32TB of sms data per 1e12 messages) in the same way that you propose that the shapes of a road and it's direction and distance "plus 1KB of metadata per meter" is enough to derive the ability to drive upon those roads. It's pretty obvious that just using sensors is not going to get FSD. Maybe in the next 20 years we will develop sensor technology (and swarm networking and whatever else) that will allow us to dispense with the "7TB" of metadata. My argument is that: we need much more "metadata" than 1KB per meter to "know the road baseline, current conditions, hazards", much in the same way a text message is more than 140 bytes. Driving with "only sensors" and rough GPS has killed people. It does not matter if human drivers have more death per million miles or whatever, because i am strictly talking about FSD, what other people are calling level 5 (i'd even concede level 4; although i wouldn't be able to use a level 4 car where i live for roughly 1/4th the year - and other areas would have more than 1/4th the year.) enjoy your night! note: the metadata for a meter of road could be: { "road_segment": { "segment_id": 3500000, "meter_position": 128534, "coordinates": { "latitude": 32.5385, "longitude": -92.9222 }, "timestamp_added": "2024-06-05T23:57:34Z", "last_updated": { "timestamp": "2025-06-05T23:57:36Z", "delta_seconds": 2.0 }, "hash_signature": "89a25b6f3cd829e671bb9d42e8fae2c6", "road_type": "highway", "lane_data": { "lane_count": 4, "lane_width_m": 3.5, "shoulder_width_m": 2.0, "divider_type": "concrete barrier", "markings": ["solid white", "dashed white", "double yellow"] }, "speed_limit_mph": 65, "road_material": "asphalt", "incline_percent": 1.2, "curve_radius_m": 150, "surface_condition": "dry", "weather_conditions": { "timestamp": "2025-06-05T23:57:36Z", "temperature_c": 25, "precipitation_mm": 0, "visibility_m": 5000, "wind_speed_mps": 2.5 }, "baseline_hazards": [ { "type": "grade_crossing", "description": "Railroad crossing with signal lights", "location": { "latitude": 32.5386, "longitude": -92.9224 } }, { "type": "roadwork", "description": "Permanent lane narrowing from past construction", "location": { "latitude": 32.5390, "longitude": -92.9230 } } ], "current_hazards": [ { "type": "construction", "description": "Active roadwork zone with lane closure", "severity": "high", "location": { "latitude": 32.5387, "longitude": -92.9226 } }, { "type": "downed_power_line", "description": "Reported electrical hazard near shoulder", "severity": "critical", "location": { "latitude": 32.5395, "longitude": -92.9232 } } ] } } Obviously you can reduce this, but there's a minimum viable amount of metadata, that's my claim, and it's more than 1KB per meter. that snippet is ~1800bytes as is. the "current conditions" would not be part of the dataset on the "7TB" disk. that would need to be relayed or otherwise ingested by the car as it drives - the way my 2012 lexus tells me that i'm about to drive into a wild storm, but that's all the extra information i get out of its infotainment system. waze is a better example of the sort of realtime updates i expect a FSD to need; although i expect many times more points of information than waze has, maybe dozens, maybe hundreds more. and each "trick" you do to reduce the size of the metadata necessarily implies more CPU needed to parse and process it.

Dylan16807 1y ago
> the most barebones amount of metadata you need to actually have actionable "intelligence" is 1KB per message (technically i was able to finagle it to ~1016 bytes.) And that's for every message, even an SMS that is the single character "K". How did you reach that number? I figure the most important metadata is source and destination phone numbers and a timestamp, and I guess what cell tower each phone was on. A phone number needs 8 bytes, and timestamp and cell tower can be 4 bytes, so that's 28 bytes of important metadata. > (quick math shows you went off something like ~32TB of sms data per 1e12 messages) I was going for a full 140TB of data. 20-30TB hard drives are available. I did consider metadata, but I figured you could probably put that in the savings from non-full-length messages. > Where are you going to put the 1PB (for 1e12, but your own cite says it's 1e13, so 10PB) of metadata, minimum? Well for just the US it would be closer to 1PB. But, uh, I'd store it in a single server rack? (ideally with backups somewhere) As of backblaze's last storage pod post, almost three years ago, it cost them $20k per petabyte. That's absolutely trivial on the scale of telecomms or governments or whatever. > My argument is that: we need much more "metadata" than 1KB per meter to "know the road baseline, current conditions, hazards", much in the same way a text message is more than 140 bytes. I mean, I agree with you about needing extra information. But that's why the number I gave is 10000x larger than your CSV. My number is supposed to be big enough to include those things! > note: the metadata for a meter of road could be: I really appreciate the effort you put into this. I have two main things to say. A) That's less than a kilobyte of information. Most of the bytes in the JSON are key names, and even without a schema for good compression, you can replace key names with 2-byte identifier numbers. And things like "critical" and "Active roadwork zone with lane closure" should also be 1-byte or 2-byte indexes into a table. And all the numbers in there could be stored as 4 byte values. Apply all that and it goes down below 300 bytes. If you had a special schema for this, it would be even lower by a significant amount. B) Most of those values would not need to be repeated per meter. Add one byte to each hazard to say how long it lasts, 0-255 meters, instant 99% savings on storing hazard data. > each "trick" you do to reduce the size of the metadata necessarily implies more CPU needed to parse and process it. CPUs are measured in billions of cycles per second. They can handle some lookup tables and basic level compression easily. Hell, these keys are just going to feed into a lookup table anyway, using integers makes it faster. And not repeating unchanged sections makes it a lot faster.

Dylan16807 1y ago

> the most barebones amount of metadata you need to actually have actionable "intelligence" is 1KB per message (technically i was able to finagle it to ~1016 bytes.) And that's for every message, even an SMS that is the single character "K". How did you reach that number? I figure the most important metadata is source and destination phone numbers and a timestamp, and I guess what cell tower each phone was on. A phone number needs 8 bytes, and timestamp and cell tower can be 4 bytes, so that's 28 bytes of important metadata. > (quick math shows you went off something like ~32TB of sms data per 1e12 messages) I was going for a full 140TB of data. 20-30TB hard drives are available. I did consider metadata, but I figured you could probably put that in the savings from non-full-length messages. > Where are you going to put the 1PB (for 1e12, but your own cite says it's 1e13, so 10PB) of metadata, minimum? Well for just the US it would be closer to 1PB. But, uh, I'd store it in a single server rack? (ideally with backups somewhere) As of backblaze's last storage pod post, almost three years ago, it cost them $20k per petabyte. That's absolutely trivial on the scale of telecomms or governments or whatever. > My argument is that: we need much more "metadata" than 1KB per meter to "know the road baseline, current conditions, hazards", much in the same way a text message is more than 140 bytes. I mean, I agree with you about needing extra information. But that's why the number I gave is 10000x larger than your CSV. My number is supposed to be big enough to include those things! > note: the metadata for a meter of road could be: I really appreciate the effort you put into this. I have two main things to say. A) That's less than a kilobyte of information. Most of the bytes in the JSON are key names, and even without a schema for good compression, you can replace key names with 2-byte identifier numbers. And things like "critical" and "Active roadwork zone with lane closure" should also be 1-byte or 2-byte indexes into a table. And all the numbers in there could be stored as 4 byte values. Apply all that and it goes down below 300 bytes. If you had a special schema for this, it would be even lower by a significant amount. B) Most of those values would not need to be repeated per meter. Add one byte to each hazard to say how long it lasts, 0-255 meters, instant 99% savings on storing hazard data. > each "trick" you do to reduce the size of the metadata necessarily implies more CPU needed to parse and process it. CPUs are measured in billions of cycles per second. They can handle some lookup tables and basic level compression easily. Hell, these keys are just going to feed into a lookup table anyway, using integers makes it faster. And not repeating unchanged sections makes it a lot faster.