3 ms·

With modern hardware, for this to be a meaningful percentage of your workload, you'd need to be working with super high data volumes (100 megabytes per second a

by csense 8y ago

With modern hardware, for this to be a meaningful percentage of your workload, you'd need to be working with super high data volumes (100 megabytes per second at the very least, maybe an order of magnitude more depending on the exact hardware). A reasonable system designer would never create an interface that accepts only hexadecimal input for bulk data transfers at those rates. That would waste half the I/O, which makes no sense! In this case, the problem with your system isn't the cycle count of your hex conversion function. It's a poorly designed interface that's forced to use hex for high data volumes. You get maybe a fraction of a percent optimizing the hex conversion function, but there's a low-hanging 100% improvement if you just use all 8 bits of your bytes. Or in Donald Knuth's words, "Premature optimization is the root of all evil."

tyingq 8y ago
Of course, you don't always get to choose your input. Feeds often come from 3rd parties or odd platforms you have no control over. I've certainly wasted lots of CPU cycles on EBCDIC to ASCII. As for this case, I assume lots of people have to parse high volume log files with things like IPV6 addresses, MAC addresses, etc. And no ability to change the things that create them.
nostrademons 8y ago
Oftentimes the biggest data-ingestion tasks happen at organizational boundaries. You find this great open-source data set that's 77TB, or you contract with another company to dump a 50GB CSV file on your FTP server every night, or you pay for API access to a feed that generates 1G/sec, or another department dumps 1TB in your S3 bucket nightly. There's a strong incentive to use standardized text-based formats (particularly JSON and CSV) for these. You can inspect sample data with a text editor, or just with 'head'. You can run really simply data analytics with just UNIX tools (I've found that a 'gunzip -c | grep | awk | grep | wc' pipeline has gotten me results in 15 seconds that would've taken 5 minutes with Python or half an hour in Java). Everybody already has a well-tested parser and serializer for the format in any language they might choose to use. Everybody's already familiar with them; you need only document field types & formats rather than giving a byte-by-byte description of the protocol. With JSON in particular, you've got 1-line parsing built into the browser instead of needing to ship a large library. If problems arise, you can dump the raw input record to the console for debugging without needing to pore over a hex dump, and you have tools available (jq, awk) for filtering out extraneous information. If you control both sides of the protocol, than by all means use an efficient binary serialization protocol like Cap'n'Proto. But a lot of interesting data ingestion problems don't fit in this category. Oftentimes the system designer doesn't actually have control of the format that data is provided to the system.

by csense 8y ago

tyingq 8y ago

Of course, you don't always get to choose your input. Feeds often come from 3rd parties or odd platforms you have no control over. I've certainly wasted lots of CPU cycles on EBCDIC to ASCII. As for this case, I assume lots of people have to parse high volume log files with things like IPV6 addresses, MAC addresses, etc. And no ability to change the things that create them.

nostrademons 8y ago

Oftentimes the biggest data-ingestion tasks happen at organizational boundaries. You find this great open-source data set that's 77TB, or you contract with another company to dump a 50GB CSV file on your FTP server every night, or you pay for API access to a feed that generates 1G/sec, or another department dumps 1TB in your S3 bucket nightly. There's a strong incentive to use standardized text-based formats (particularly JSON and CSV) for these. You can inspect sample data with a text editor, or just with 'head'. You can run really simply data analytics with just UNIX tools (I've found that a 'gunzip -c | grep | awk | grep | wc' pipeline has gotten me results in 15 seconds that would've taken 5 minutes with Python or half an hour in Java). Everybody already has a well-tested parser and serializer for the format in any language they might choose to use. Everybody's already familiar with them; you need only document field types & formats rather than giving a byte-by-byte description of the protocol. With JSON in particular, you've got 1-line parsing built into the browser instead of needing to ship a large library. If problems arise, you can dump the raw input record to the console for debugging without needing to pore over a hex dump, and you have tools available (jq, awk) for filtering out extraneous information. If you control both sides of the protocol, than by all means use an efficient binary serialization protocol like Cap'n'Proto. But a lot of interesting data ingestion problems don't fit in this category. Oftentimes the system designer doesn't actually have control of the format that data is provided to the system.