3 ms·

> I've never found text-based formats to be a good choice for data that humans will rarely need to read or write; A computer may need to read the data millions

> I've never found text-based formats to be a good choice for data that humans will rarely need to read or write; A computer may need to read the data millions of times, but humans only need to read it when things goes wrong. If you take the route of using a binary format, then you need to provide a robust set of tools to view, edit, debug that format. It can absolutely work, journald is binary, but has the tooling to help you get the plain text, when you need it. Protobuf could work well, but if you design your own binary data format you have to build some tooling as well. For many (most) it doesn't matter much if you use XML or JSON, the computer will process it fast enough. If the concern is parsing speed, then yes, absolutely binary formats might be the best option. For things like HFT or anything else that processes vast volumes of data, its probably the only choice.

userbinator 3y ago
I can read from a hexdump a few binary formats I've worked with over the years. Not really harder than learning any other language, but much more concise than text-based formats.
- mrweasel 3y ago
  Given that I don't know what you do, that might be a completely reasonable solution, in your case. My background is being woken up at 3AM by alert of call for systems where I have less that 100% familiarity and is asked to troubleshot. Looking at a hexdump for a format I don't know is not all that helpful. Seeming a mangled XML document, or reading the JSON output from an API call can frequently be enough to know if I need to call a developer, or if I can work with the client and fix the data. Binary formats are absolutely a valid solution of a number of problems and if it works in your context there's no reason to change it.
anonzzzies 3y ago
For most things, SQLite is a very good format and somehow still underused in favour of custom formats (without or with very basic) tooling.
- fmajid 3y ago
  They do say their competition is fwrite(), not MySQL or PostgreSQL, just like Coca-Cola says its competition is water, not Pepsi…
Sn0wCoder 3y ago
Gotta agrees here. How many times a day do you get asked what data did your API send to my API? For me it’s daily. Need a format that is logged and able to be displayed in an admin UI somewhere, so a DB query is not needed to prove yes A was sent to you so if A is not in your system, then it’s an issue on the other side. I would not want to debug an API that talked to another API in binary…. I mean XML or JSON is still better than binary… I love applying for jobs that ask if I know JASON because the answer is always yes when its XML I don’t know anyone by that name
- vineyardmike 3y ago
  I agree with the high level point, but I’m not sure I have similar experiences. No system I’ve worked on has ever directly logged over-the-wire bytes (for obvious privacy reasons), so really the protocol can be binary. In fact, for years now most of my work has been with ProtoBufs, which has amazing tooling and an oft-forgotten text format for us mortals.
  - Sn0wCoder 3y ago
    Sorry for the confusion, it’s not like a text log on the server. API A is calling API B. API B grabs the data from the data store, before sending to API A – API B logs the JSON in the same data store that the data is coming from. When end user of API A says did you send me the right info, end user in API A says yes I can see the data being sent via UI. I mean yes this could all be ONES and ZEROS but nice to let some admin user prove to other user that data was sent VIA JSON object that API A accepts. IDK they are all ONES and ZEROS some are just easier for end user to grok.
- novagameco 3y ago
  At my last company we had billions of lines of logs to go through every day. Most logs were in an ELB (Elastic Load Balancer; basically each log was a set of HTTP headers and GET params) format which were dumped to S3. We then used Athena queries to build a timeline of events as they passed through our various microservices. The benefit of Athena is that it's insanely fast to mount on top of S3 logs since it's AWS->AWS and you don't have to download hundreds of megabytes of logs to do your queries
michaeljx 3y ago
Even in HFT, they use a text based API (FIX)
- c-cube 3y ago
  FIX is old and really not great. Some exchanges are trying to move, or have moved, to things like SBE which is a zero-copy binary format. Of course it also improves performance dramatically. So maybe that's not the best example :-)

userbinator 3y ago

I can read from a hexdump a few binary formats I've worked with over the years. Not really harder than learning any other language, but much more concise than text-based formats.

mrweasel 3y ago
Given that I don't know what you do, that might be a completely reasonable solution, in your case. My background is being woken up at 3AM by alert of call for systems where I have less that 100% familiarity and is asked to troubleshot. Looking at a hexdump for a format I don't know is not all that helpful. Seeming a mangled XML document, or reading the JSON output from an API call can frequently be enough to know if I need to call a developer, or if I can work with the client and fix the data. Binary formats are absolutely a valid solution of a number of problems and if it works in your context there's no reason to change it.

anonzzzies 3y ago

For most things, SQLite is a very good format and somehow still underused in favour of custom formats (without or with very basic) tooling.

fmajid 3y ago
They do say their competition is fwrite(), not MySQL or PostgreSQL, just like Coca-Cola says its competition is water, not Pepsi…

Sn0wCoder 3y ago

Gotta agrees here. How many times a day do you get asked what data did your API send to my API? For me it’s daily. Need a format that is logged and able to be displayed in an admin UI somewhere, so a DB query is not needed to prove yes A was sent to you so if A is not in your system, then it’s an issue on the other side. I would not want to debug an API that talked to another API in binary…. I mean XML or JSON is still better than binary… I love applying for jobs that ask if I know JASON because the answer is always yes when its XML I don’t know anyone by that name

vineyardmike 3y ago
I agree with the high level point, but I’m not sure I have similar experiences. No system I’ve worked on has ever directly logged over-the-wire bytes (for obvious privacy reasons), so really the protocol can be binary. In fact, for years now most of my work has been with ProtoBufs, which has amazing tooling and an oft-forgotten text format for us mortals.
- Sn0wCoder 3y ago
  Sorry for the confusion, it’s not like a text log on the server. API A is calling API B. API B grabs the data from the data store, before sending to API A – API B logs the JSON in the same data store that the data is coming from. When end user of API A says did you send me the right info, end user in API A says yes I can see the data being sent via UI. I mean yes this could all be ONES and ZEROS but nice to let some admin user prove to other user that data was sent VIA JSON object that API A accepts. IDK they are all ONES and ZEROS some are just easier for end user to grok.
novagameco 3y ago
At my last company we had billions of lines of logs to go through every day. Most logs were in an ELB (Elastic Load Balancer; basically each log was a set of HTTP headers and GET params) format which were dumped to S3. We then used Athena queries to build a timeline of events as they passed through our various microservices. The benefit of Athena is that it's insanely fast to mount on top of S3 logs since it's AWS->AWS and you don't have to download hundreds of megabytes of logs to do your queries

michaeljx 3y ago

Even in HFT, they use a text based API (FIX)

c-cube 3y ago
FIX is old and really not great. Some exchanges are trying to move, or have moved, to things like SBE which is a zero-copy binary format. Of course it also improves performance dramatically. So maybe that's not the best example :-)