3 ms·

Gotta agrees here. How many times a day do you get asked what data did your API send to my API? For me it’s daily. Need a format that is logged and able to b

Gotta agrees here. How many times a day do you get asked what data did your API send to my API? For me it’s daily. Need a format that is logged and able to be displayed in an admin UI somewhere, so a DB query is not needed to prove yes A was sent to you so if A is not in your system, then it’s an issue on the other side. I would not want to debug an API that talked to another API in binary…. I mean XML or JSON is still better than binary… I love applying for jobs that ask if I know JASON because the answer is always yes when its XML I don’t know anyone by that name

vineyardmike 3y ago
I agree with the high level point, but I’m not sure I have similar experiences. No system I’ve worked on has ever directly logged over-the-wire bytes (for obvious privacy reasons), so really the protocol can be binary. In fact, for years now most of my work has been with ProtoBufs, which has amazing tooling and an oft-forgotten text format for us mortals.
- Sn0wCoder 3y ago
  Sorry for the confusion, it’s not like a text log on the server. API A is calling API B. API B grabs the data from the data store, before sending to API A – API B logs the JSON in the same data store that the data is coming from. When end user of API A says did you send me the right info, end user in API A says yes I can see the data being sent via UI. I mean yes this could all be ONES and ZEROS but nice to let some admin user prove to other user that data was sent VIA JSON object that API A accepts. IDK they are all ONES and ZEROS some are just easier for end user to grok.
novagameco 3y ago
At my last company we had billions of lines of logs to go through every day. Most logs were in an ELB (Elastic Load Balancer; basically each log was a set of HTTP headers and GET params) format which were dumped to S3. We then used Athena queries to build a timeline of events as they passed through our various microservices. The benefit of Athena is that it's insanely fast to mount on top of S3 logs since it's AWS->AWS and you don't have to download hundreds of megabytes of logs to do your queries

vineyardmike 3y ago

I agree with the high level point, but I’m not sure I have similar experiences. No system I’ve worked on has ever directly logged over-the-wire bytes (for obvious privacy reasons), so really the protocol can be binary. In fact, for years now most of my work has been with ProtoBufs, which has amazing tooling and an oft-forgotten text format for us mortals.

Sn0wCoder 3y ago
Sorry for the confusion, it’s not like a text log on the server. API A is calling API B. API B grabs the data from the data store, before sending to API A – API B logs the JSON in the same data store that the data is coming from. When end user of API A says did you send me the right info, end user in API A says yes I can see the data being sent via UI. I mean yes this could all be ONES and ZEROS but nice to let some admin user prove to other user that data was sent VIA JSON object that API A accepts. IDK they are all ONES and ZEROS some are just easier for end user to grok.

novagameco 3y ago

At my last company we had billions of lines of logs to go through every day. Most logs were in an ELB (Elastic Load Balancer; basically each log was a set of HTTP headers and GET params) format which were dumped to S3. We then used Athena queries to build a timeline of events as they passed through our various microservices. The benefit of Athena is that it's insanely fast to mount on top of S3 logs since it's AWS->AWS and you don't have to download hundreds of megabytes of logs to do your queries