18 ms·

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doin

People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware. Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade & order book feeds from the top 10 exchanges. It handled about 3K incoming messages/second (~260M per day), including all of the message parsing, order book update, processing, streaming to websocket connections on any connected client, and archival to PostGres for historical processing. Total hardware required was 1 m4.large + 1 r5.large AWS instances, for a bit under $200/month, and the boxes would regularly run at about 50% CPU.

fizwhiz 5y ago
Would you mind describing your stack in more detail? Did you use gRPC with Go?
- nostrademons 5y ago
  Sure, startup is defunct now and I think arbitrage & data on centralized exchanges is a dead market now. Wall Street HFTs got into the arbitrage game, and the data sites laypeople actually visit are the ones started in 2014. Codebase was pure server-side Kotlin running on the JVM. Jackson for JSON parsing, when the exchange didn't provide their own client library (I used the native client libraries when they did). Think I used Undertow for exchange websockets, and Jetty for webserving & client websockets. Postgres for DB. The threading model was actually the biggest bottleneck, and took a few tries to get right. I did JSON parsing and conversion to a common representation on the incoming IO thread. Then everything would get dumped into a big producer/consumer queue, and picked up by a per-CPU threadpool. Main thread handled price normalization (many crypto assets don't trade in USD, so you have to convert through BTC/ETH/USDT to get dollar prices), order book update, volume computations, opportunity detection, and other business logic. It also compared timestamps on incoming messages, and each new second, it'd aggregate the messages for that second (I only cared about historical data on a 1s basis) and hand them off to a separate DB thread. DB would do a big bulk insert every second; this is how I kept database writes below Postgres's QPS limit. Client websocket connections were handled internally within Jetty, which I think uses a threadpool and NIO. Key architectural principles were 1) do everything in RAM - the RDS machine was the only one that touched disk, and writes to it were strictly throttled 2) throw away data as soon as you're done with it - I had a bunch of OOM issues by trying to put unparsed messages in the main producer/consumer queue rather than parsing and discarding them 3) aggregate & compute early - keep final requirements in mind and don't save raw data you don't need 4) separate blocking and non-blocking activities on different threads, preferring non-blocking whenever possible and 5) limit threads to only those activities that are actively doing work.
  - ochoseis 5y ago
    Would you use Kotlin again for the back end? Having not yet used it for that purpose, it seems like you’d get the benefit of the JVM ecosystem along with a nice language (but perhaps too many power-features).
    - nostrademons 5y ago
      Yes. Kotlin is great for both front and backend. For that particular use-case (or related financial ones) I'd consider Rust, which was a bit too immature when I was working on this but would give you some extra speed. HFT is winner-take-all and the bar has risen significantly even in the last couple years, so if I were putting my own money at risk now I'd want the absolute fastest processing possible.
  - mikhmha 5y ago
    i tried to do something similar recently. Your architecture sounds a lot like mine. I did all the thread pool management and JSON parsing using C++ libraries/frameworks. The real-time data was visualized on a website. Here is an example. https://algot.io/ https://algot.io/
Gepsens 5y ago
I'm running a crypto trading platform I'm developing on 30$ on DigitalOcean. I coded exclusively in Rust and recently added a dynamic interface to python. Today during the BTC crash it spiked at 20k events/s, and that's only incoming data.
- giancarlostoro 5y ago
  is that $30 for a single droplet or is it spread out between a few different services? I'm kind of curious since I use DO for small projects myself.
  - bochoh 5y ago
    I’m also curious on this. My stack currently is a SQLite file -> ORM -> .net core on Linux on a single box
  - Gepsens 5y ago
    It's not a single droplet, it's 3 + data
  - Gepsens 5y ago
    I use terraform and ansible, one droplet has monitoring (prom and grafana), 2 run services that currently run without any other infrastructure The rest of the cost is a 250 gb space on DO
- danudey 5y ago
  > I coded exclusively in Rust This reminds me of back in 2003, a friend of mine worked for an online casino vendor; basically, if you wanted to run an online casino, you'd buy the software from a company and customize it to fit your theme. They were often written in Java, ASP.NET, and so on. They were extremely heavyweight. They'd need 8-10 servers for 10k users. They hogged huge amounts of RAM. My friend wrote the one this company was selling in C. Not even C++, mind you, just C. The game modules were chosen at compile time, so unwanted games didn't exist. The entire binary (as in, 100% of the code) compiled to just over 3 MB when stripped. He could handle 10k concurrent users on one single-core server. I'm never gonna stop writing things in Python, but it still amazes me what can happen when you get down close to the metal.
fhood 5y ago
People may underestimate how fast modern machines are, but that is probably in part because, at least in my fairly relevant experience, I have literally never seen a CPU bottleneck under normal circumstances. Memory pressure is nearly always the driving issue.
- foobarbazetc 5y ago
  Yeah. Now that CPUs are insanely powerful and you have NVMe SSDs etc the bottleneck is always memory.
  - zozbot234 5y ago
    Sure, but as far as software is concerned, optimizing for memory bandwidth (the typical bottleneck in modern systems) is not so different from optimizing for CPU.
  - nostrademons 5y ago
    It's also amazing how much you can fit in RAM if you're careful. I remember ~2007 people were aghast at Facebook's 4T memcached deployment that stored basically everyone's social network posts; now you can get single servers for ~$4K with 4T of RAM. The trick is basically that you have to eschew the last 15 years of "productivity" enhancements. Pretty much any dynamic language is out; if you must use the JVM or .NET, store as much as possible in flat buffers of primitive types. I ended up converting order books from the obvious representation (hashtable mapping prices to a list of Order structs) to a pair of SortedMaps from FastUtils, which provides an unboxed float representation with no pointers. That change ended up reducing memory usage by about 4x. You can fit a lot of ints and floats in today's 100G+ machines, way more than needed to represent the entire cryptocurrency market. You just can't do that when you're chasing 3 pointers, each with their associated object headers, to store 4 bytes.
    - NetToolKit 5y ago
      > now you can get single servers for ~$4K with 4T of RAM Does the $4K include the cost of the RAM? Where can I find these servers? Thanks!
danudey 5y ago
A lot of that is due to absolutely lousy code. We had a system management backend at my last company. Loading the users list was unbearably slow; 10+ seconds on a warm cache. Not too terrible, except that most user management tasks required a page reload, so it was just wildly infuriating. Eventually I took a look at the code for the page, which queried LDAP for user data and the database for permissions data. It did: get list of users foreach user: get list of all permissions filter down to the ones assigned directly to the user foreach user: get list of all groups foreach group: get list of all permissions filter down to the ones assigned to the group filter down to the ones the user has I'm no algorithm genius, but I'm pretty sure O(n^2+n^3) is not an efficient one. I replaced it with get list of all users get list of all groups get list of all permissions <filter accordingly> Suffice to say, it was a lot more responsive. Also worth noting was that fetching the user list required shelling out to a command (a python script) which shelled out to a command (ldapsearch), and the whole system was a nightmare. There were also dozens of pages where almost no processing was done in the view, but a bunch of objects with lazy-loaded properties were passed into the template and always used, so when benchmarking you'd get 0.01 seconds for the entire function and then 233 seconds for "return render(...)' because for every single row in the database (dozens or hundreds) the template would access a property that would trigger another SQL call to the backend, rather than just doing one giant "SELECT ALL THE THINGS" and hammering it out that way. Note that we also weren't using Django's foreign keys support, so we couldn't even tell Django to "fetch everything non-lazily" because it had no idea. If that app were written right it could have run on a Raspberry Pi 2, but instead there was no amount of cores that could have sped it up.
- JJMcJ 5y ago
  Your pattern is quite powerful: get data from several sources and do the rearranging on the client (which might be a web server), instead of multiple interactions for each data item. For SQL you can also do a stored procedure. Sometimes that works well if you are good at your DBMS's procedure language and the schema is good.
  - spfzero 5y ago
dirkg 5y ago
may be OT, but how do you subscribe to these trade feeds, is there a unified service or do you need to do it individually for each source, and how much does it cost approximately ? I'm guessing if you put all this data into Kinesis or message queues it would end up costing quite a bit more.
- meltedcapacitor 5y ago
  Never heard of a crypto exchange that charges for data feeds, the norm is free and fast. One of the positive of the industry compared to old school finance. They're rent seeking in other ways though, no worries.
- nostrademons 5y ago
  There are probably unified services that let you do it - I was kinda competing in this area but didn't want to deal with enterprise sales, and it's a bit of a hard sell anyway. If you do it individually, there are public developer docs for each exchange that explain how their API works. It's generally free as long as you're not making a large number of active trades.
blacklion 5y ago
Crypto markets are very small :-) I'm working and company which process "real" exchanges, like NASDAQ, LSE, and, especially, OPRA feed. We've added 20+ crypto exchanges in our portfolio this year, and all of them are processed on one old server which is unable to process NASDAQ Total View in real-time anymore. On the other hand, whole OPRA feed (more than 5Gbit/s or 65B/day, yes, it is billions, messages of very optimized binary protocol, not this crappy JSON) is processed by our code on one modern server. Nothing special, two sockets of Intel Xeons (not even Platinums).
- joering2 5y ago
  Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.
  - paraph1n 5y ago
    There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers
    - rewq4321 5y ago
      And msgpack if you want an order of magnitude faster serialization/deserialisation and can put up with worse compression (I think mainly due to schema overhead since protobuf files don't store the schema?) https://msgpack.org/index.html https://msgpack.org/index.html Good protobuf vs msgpack comparison: https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/@hugovs/the-need-for-speed-experimenting-...
    - blacklion 5y ago
      "old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols. Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST. I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds). I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes. Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.
mynameisash 5y ago
What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I've worked with a few different systems that compile their queries into C/C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to process. I get that quick & dirty scripts for exploration don't need to be super optimized, and that throwing more hardware at the problem _can_ be cheaper than engineering time, but in my experience, the latter ends up costing my org tens of millions of dollars annually -- just write some code and allocate a ton of resources to make it work in a reasonable amount of time. I'm hopeful that Ballista[1], for example, will see uptake and improve this. [0] https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law [1] https://github.com/apache/arrow-datafusion/tree/master/ballista https://github.com/apache/arrow-datafusion/tree/master/balli...
- b9a2cab5 5y ago
  Abstractions almost always end up leaky. Spark SQL, for example, does whole-stage codegen which collapses multiple project/filter stages into a single compiled stage, but your underlying data format still needs to be memory friendly (i.e. linear accesses, low branching, etc.). The codegen is very naive and the JVM JIT can only do so much. What I've seen is that you need people who deeply understand the system (e.g. Spark) to be able to tune for these edge cases (e.g. see [1] for examples of some of the tradeoffs between different processing schemes). Those people are expensive (think $500k+ annual salaries) and are really only cost effective when your compute spend is in the tens of millions or higher annually. Everyone else is using open source and throwing more compute at the problem or relying on their data scientists/data engineers to figure out what magic knob to turn. [1]: https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf
  - paulluuk 5y ago
    What's wrong with relying on data engineers for data engineering?
arjie 5y ago
How do I get in touch with you? Definitely using more resources than this to process fewer integrations. I’m curious what trade offs you made to enable this. If you’re anywhere in the US, let me know.
aoms 5y ago
How much bandwidth did this use daily/monthly?
- nostrademons 5y ago
  I don't remember offhand. I think bandwidth costs averaged about $40-50 out of that ~$200/month.
fukmbas 5y ago
Turns out cloud iaas is severely overpriced and it will always be cheaper to run on prem...let's see how long this lasts
lewisjoe 5y ago
I'm running two different projects on a single instance of cheap VM too. Both of them runs on a not-so-memory-efficient programming runtime, yet the VM handles the load just fine. Of course, a lot of it depends on what your app does for each request but most apps are simple enough and can live with being a monolith / single fat binary running on a single instance. The problem with today's DevOps culture is that they present K8's as answers for everything. Instead of defining a clear line on when to use them and when not to.
ingas 5y ago
In 2001 we started a GSM-operator on Compaq Server (it was before they were bought by HP) with whole 1Gb(!) of RAM and 2x10Gb SCSI disks. It served up to 70K of subscribers, call center with 30-40 employees, payment systems integration, everything. Next was 8 socket Intel server. We were never able to saturate it's CPUs - 300 Mhz (or was it 400 ?) bus was a stopper. It served 350-400K of subscribers. And next: we changed architecture and used 2 servers with 2 socket Intel CPUs again but that was time when Ghz frequencies appeared on market. We dreamed about 4xAMD server. We came to ~1 mln of active subscribers. Nowadays: every phone has more power than it was those servers. Typical react application consumes more resources than billing system. Gigabyte here, gigabyte there - nobody counts them. /grumpy oldster mode
Cthulhu_ 5y ago
They really do, and because of that they reach for over-engineered infrastructure solutions. I mean I get that you'd like to have some redundancy for your webserver and database, maybe some DDOS mitigation, off-site backup, etc, but you don't need an overblown microservices architecture for a low-traffic CRUD application. That just creates problems, instead of solving problems you WISH you had, and it's slower than starting simple.
rattray 5y ago
What were the most important aspects of the technology stack which enabled that?
Breza 5y ago
I totally agree! I run a data science department at a corporation and it's amazing how much of our job is done on our laptops. I have a Dell Precision. When I need more power (a very rare situation), I can spin up a GPU cloud server and complete my big analysis for under $5.
milkywaybrain 5y ago
True. I can completely relate with this. I developed an open source crypto exchange connector [0] and created a fun twitter bot [1] on top of that. Currently twitter bot processes all the USDT market trades from Binance (around 260 markets with average 30,000 trades per minute) and calculates OHLC metrics every 15 minutes using InfluxDB. All these installations and calculations are done in a free tier 1 VCPU / 1 GB RAM AWS server (less than 10% CPU and less than 40% RAM usage always). [0] : https://github.com/milkywaybrain/cryptogalaxy https://github.com/milkywaybrain/cryptogalaxy [1] : https://twitter.com/moon_or_earth https://twitter.com/moon_or_earth

fizwhiz 5y ago

Would you mind describing your stack in more detail? Did you use gRPC with Go?

nostrademons 5y ago
Sure, startup is defunct now and I think arbitrage & data on centralized exchanges is a dead market now. Wall Street HFTs got into the arbitrage game, and the data sites laypeople actually visit are the ones started in 2014. Codebase was pure server-side Kotlin running on the JVM. Jackson for JSON parsing, when the exchange didn't provide their own client library (I used the native client libraries when they did). Think I used Undertow for exchange websockets, and Jetty for webserving & client websockets. Postgres for DB. The threading model was actually the biggest bottleneck, and took a few tries to get right. I did JSON parsing and conversion to a common representation on the incoming IO thread. Then everything would get dumped into a big producer/consumer queue, and picked up by a per-CPU threadpool. Main thread handled price normalization (many crypto assets don't trade in USD, so you have to convert through BTC/ETH/USDT to get dollar prices), order book update, volume computations, opportunity detection, and other business logic. It also compared timestamps on incoming messages, and each new second, it'd aggregate the messages for that second (I only cared about historical data on a 1s basis) and hand them off to a separate DB thread. DB would do a big bulk insert every second; this is how I kept database writes below Postgres's QPS limit. Client websocket connections were handled internally within Jetty, which I think uses a threadpool and NIO. Key architectural principles were 1) do everything in RAM - the RDS machine was the only one that touched disk, and writes to it were strictly throttled 2) throw away data as soon as you're done with it - I had a bunch of OOM issues by trying to put unparsed messages in the main producer/consumer queue rather than parsing and discarding them 3) aggregate & compute early - keep final requirements in mind and don't save raw data you don't need 4) separate blocking and non-blocking activities on different threads, preferring non-blocking whenever possible and 5) limit threads to only those activities that are actively doing work.
- ochoseis 5y ago
  Would you use Kotlin again for the back end? Having not yet used it for that purpose, it seems like you’d get the benefit of the JVM ecosystem along with a nice language (but perhaps too many power-features).
  - nostrademons 5y ago
    Yes. Kotlin is great for both front and backend. For that particular use-case (or related financial ones) I'd consider Rust, which was a bit too immature when I was working on this but would give you some extra speed. HFT is winner-take-all and the bar has risen significantly even in the last couple years, so if I were putting my own money at risk now I'd want the absolute fastest processing possible.
- mikhmha 5y ago
  i tried to do something similar recently. Your architecture sounds a lot like mine. I did all the thread pool management and JSON parsing using C++ libraries/frameworks. The real-time data was visualized on a website. Here is an example. https://algot.io/ https://algot.io/

Gepsens 5y ago

I'm running a crypto trading platform I'm developing on 30$ on DigitalOcean. I coded exclusively in Rust and recently added a dynamic interface to python. Today during the BTC crash it spiked at 20k events/s, and that's only incoming data.

giancarlostoro 5y ago
is that $30 for a single droplet or is it spread out between a few different services? I'm kind of curious since I use DO for small projects myself.
- bochoh 5y ago
  I’m also curious on this. My stack currently is a SQLite file -> ORM -> .net core on Linux on a single box
- Gepsens 5y ago
  It's not a single droplet, it's 3 + data
- Gepsens 5y ago
  I use terraform and ansible, one droplet has monitoring (prom and grafana), 2 run services that currently run without any other infrastructure The rest of the cost is a 250 gb space on DO
danudey 5y ago
> I coded exclusively in Rust This reminds me of back in 2003, a friend of mine worked for an online casino vendor; basically, if you wanted to run an online casino, you'd buy the software from a company and customize it to fit your theme. They were often written in Java, ASP.NET, and so on. They were extremely heavyweight. They'd need 8-10 servers for 10k users. They hogged huge amounts of RAM. My friend wrote the one this company was selling in C. Not even C++, mind you, just C. The game modules were chosen at compile time, so unwanted games didn't exist. The entire binary (as in, 100% of the code) compiled to just over 3 MB when stripped. He could handle 10k concurrent users on one single-core server. I'm never gonna stop writing things in Python, but it still amazes me what can happen when you get down close to the metal.

fhood 5y ago

People may underestimate how fast modern machines are, but that is probably in part because, at least in my fairly relevant experience, I have literally never seen a CPU bottleneck under normal circumstances. Memory pressure is nearly always the driving issue.

foobarbazetc 5y ago
Yeah. Now that CPUs are insanely powerful and you have NVMe SSDs etc the bottleneck is always memory.
- zozbot234 5y ago
  Sure, but as far as software is concerned, optimizing for memory bandwidth (the typical bottleneck in modern systems) is not so different from optimizing for CPU.
- nostrademons 5y ago
  It's also amazing how much you can fit in RAM if you're careful. I remember ~2007 people were aghast at Facebook's 4T memcached deployment that stored basically everyone's social network posts; now you can get single servers for ~$4K with 4T of RAM. The trick is basically that you have to eschew the last 15 years of "productivity" enhancements. Pretty much any dynamic language is out; if you must use the JVM or .NET, store as much as possible in flat buffers of primitive types. I ended up converting order books from the obvious representation (hashtable mapping prices to a list of Order structs) to a pair of SortedMaps from FastUtils, which provides an unboxed float representation with no pointers. That change ended up reducing memory usage by about 4x. You can fit a lot of ints and floats in today's 100G+ machines, way more than needed to represent the entire cryptocurrency market. You just can't do that when you're chasing 3 pointers, each with their associated object headers, to store 4 bytes.
  - NetToolKit 5y ago
    > now you can get single servers for ~$4K with 4T of RAM Does the $4K include the cost of the RAM? Where can I find these servers? Thanks!

danudey 5y ago

A lot of that is due to absolutely lousy code. We had a system management backend at my last company. Loading the users list was unbearably slow; 10+ seconds on a warm cache. Not too terrible, except that most user management tasks required a page reload, so it was just wildly infuriating. Eventually I took a look at the code for the page, which queried LDAP for user data and the database for permissions data. It did: get list of users foreach user: get list of all permissions filter down to the ones assigned directly to the user foreach user: get list of all groups foreach group: get list of all permissions filter down to the ones assigned to the group filter down to the ones the user has I'm no algorithm genius, but I'm pretty sure O(n^2+n^3) is not an efficient one. I replaced it with get list of all users get list of all groups get list of all permissions <filter accordingly> Suffice to say, it was a lot more responsive. Also worth noting was that fetching the user list required shelling out to a command (a python script) which shelled out to a command (ldapsearch), and the whole system was a nightmare. There were also dozens of pages where almost no processing was done in the view, but a bunch of objects with lazy-loaded properties were passed into the template and always used, so when benchmarking you'd get 0.01 seconds for the entire function and then 233 seconds for "return render(...)' because for every single row in the database (dozens or hundreds) the template would access a property that would trigger another SQL call to the backend, rather than just doing one giant "SELECT ALL THE THINGS" and hammering it out that way. Note that we also weren't using Django's foreign keys support, so we couldn't even tell Django to "fetch everything non-lazily" because it had no idea. If that app were written right it could have run on a Raspberry Pi 2, but instead there was no amount of cores that could have sped it up.

JJMcJ 5y ago
Your pattern is quite powerful: get data from several sources and do the rearranging on the client (which might be a web server), instead of multiple interactions for each data item. For SQL you can also do a stored procedure. Sometimes that works well if you are good at your DBMS's procedure language and the schema is good.
- spfzero 5y ago

dirkg 5y ago

may be OT, but how do you subscribe to these trade feeds, is there a unified service or do you need to do it individually for each source, and how much does it cost approximately ? I'm guessing if you put all this data into Kinesis or message queues it would end up costing quite a bit more.

meltedcapacitor 5y ago
Never heard of a crypto exchange that charges for data feeds, the norm is free and fast. One of the positive of the industry compared to old school finance. They're rent seeking in other ways though, no worries.
nostrademons 5y ago
There are probably unified services that let you do it - I was kinda competing in this area but didn't want to deal with enterprise sales, and it's a bit of a hard sell anyway. If you do it individually, there are public developer docs for each exchange that explain how their API works. It's generally free as long as you're not making a large number of active trades.

blacklion 5y ago

Crypto markets are very small :-) I'm working and company which process "real" exchanges, like NASDAQ, LSE, and, especially, OPRA feed. We've added 20+ crypto exchanges in our portfolio this year, and all of them are processed on one old server which is unable to process NASDAQ Total View in real-time anymore. On the other hand, whole OPRA feed (more than 5Gbit/s or 65B/day, yes, it is billions, messages of very optimized binary protocol, not this crappy JSON) is processed by our code on one modern server. Nothing special, two sockets of Intel Xeons (not even Platinums).

joering2 5y ago
Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.
- paraph1n 5y ago
  There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers
  - rewq4321 5y ago
    And msgpack if you want an order of magnitude faster serialization/deserialisation and can put up with worse compression (I think mainly due to schema overhead since protobuf files don't store the schema?) https://msgpack.org/index.html https://msgpack.org/index.html Good protobuf vs msgpack comparison: https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/@hugovs/the-need-for-speed-experimenting-...
  - blacklion 5y ago
    "old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols. Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST. I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds). I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes. Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.

mynameisash 5y ago

What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I've worked with a few different systems that compile their queries into C/C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to process. I get that quick & dirty scripts for exploration don't need to be super optimized, and that throwing more hardware at the problem _can_ be cheaper than engineering time, but in my experience, the latter ends up costing my org tens of millions of dollars annually -- just write some code and allocate a ton of resources to make it work in a reasonable amount of time. I'm hopeful that Ballista[1], for example, will see uptake and improve this. [0] https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law [1] https://github.com/apache/arrow-datafusion/tree/master/ballista https://github.com/apache/arrow-datafusion/tree/master/balli...

b9a2cab5 5y ago
Abstractions almost always end up leaky. Spark SQL, for example, does whole-stage codegen which collapses multiple project/filter stages into a single compiled stage, but your underlying data format still needs to be memory friendly (i.e. linear accesses, low branching, etc.). The codegen is very naive and the JVM JIT can only do so much. What I've seen is that you need people who deeply understand the system (e.g. Spark) to be able to tune for these edge cases (e.g. see [1] for examples of some of the tradeoffs between different processing schemes). Those people are expensive (think $500k+ annual salaries) and are really only cost effective when your compute spend is in the tens of millions or higher annually. Everyone else is using open source and throwing more compute at the problem or relying on their data scientists/data engineers to figure out what magic knob to turn. [1]: https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf
- paulluuk 5y ago
  What's wrong with relying on data engineers for data engineering?

arjie 5y ago

How do I get in touch with you? Definitely using more resources than this to process fewer integrations. I’m curious what trade offs you made to enable this. If you’re anywhere in the US, let me know.

aoms 5y ago

How much bandwidth did this use daily/monthly?

nostrademons 5y ago
I don't remember offhand. I think bandwidth costs averaged about $40-50 out of that ~$200/month.

fukmbas 5y ago

Turns out cloud iaas is severely overpriced and it will always be cheaper to run on prem...let's see how long this lasts

lewisjoe 5y ago

I'm running two different projects on a single instance of cheap VM too. Both of them runs on a not-so-memory-efficient programming runtime, yet the VM handles the load just fine. Of course, a lot of it depends on what your app does for each request but most apps are simple enough and can live with being a monolith / single fat binary running on a single instance. The problem with today's DevOps culture is that they present K8's as answers for everything. Instead of defining a clear line on when to use them and when not to.

ingas 5y ago

In 2001 we started a GSM-operator on Compaq Server (it was before they were bought by HP) with whole 1Gb(!) of RAM and 2x10Gb SCSI disks. It served up to 70K of subscribers, call center with 30-40 employees, payment systems integration, everything. Next was 8 socket Intel server. We were never able to saturate it's CPUs - 300 Mhz (or was it 400 ?) bus was a stopper. It served 350-400K of subscribers. And next: we changed architecture and used 2 servers with 2 socket Intel CPUs again but that was time when Ghz frequencies appeared on market. We dreamed about 4xAMD server. We came to ~1 mln of active subscribers. Nowadays: every phone has more power than it was those servers. Typical react application consumes more resources than billing system. Gigabyte here, gigabyte there - nobody counts them. /grumpy oldster mode

Cthulhu_ 5y ago

They really do, and because of that they reach for over-engineered infrastructure solutions. I mean I get that you'd like to have some redundancy for your webserver and database, maybe some DDOS mitigation, off-site backup, etc, but you don't need an overblown microservices architecture for a low-traffic CRUD application. That just creates problems, instead of solving problems you WISH you had, and it's slower than starting simple.

rattray 5y ago

What were the most important aspects of the technology stack which enabled that?

Breza 5y ago

I totally agree! I run a data science department at a corporation and it's amazing how much of our job is done on our laptops. I have a Dell Precision. When I need more power (a very rare situation), I can spin up a GPU cloud server and complete my big analysis for under $5.

milkywaybrain 5y ago

True. I can completely relate with this. I developed an open source crypto exchange connector [0] and created a fun twitter bot [1] on top of that. Currently twitter bot processes all the USDT market trades from Binance (around 260 markets with average 30,000 trades per minute) and calculates OHLC metrics every 15 minutes using InfluxDB. All these installations and calculations are done in a free tier 1 VCPU / 1 GB RAM AWS server (less than 10% CPU and less than 40% RAM usage always). [0] : https://github.com/milkywaybrain/cryptogalaxy https://github.com/milkywaybrain/cryptogalaxy [1] : https://twitter.com/moon_or_earth https://twitter.com/moon_or_earth