3 ms·
This is a thoroughly interesting topic. Thanks for the article. I haven't thought much about iterators link to coroutines. As a hobby, I am working to write a
by samsquire 3y ago
This is a thoroughly interesting topic. Thanks for the article.
I haven't thought much about iterators link to coroutines.
As a hobby, I am working to write about a dream programming language. I happen to be really interested in parallelism, asynchronous, coroutines, multithreading and concurrency.
I want:
* seamlessly switch between remote-thread coroutine, local thread coroutine.
* concurrency and parallelism and async to be easy to think about, reason about, read and program
* programs should be easy to parallelise and be async and concurrent
Go iterators seem to be local to a thread, but what if you want to distribute work across threads?
I've been thinking of scheduling recently.
Imagine you're a search engine company and you want to index links between URLs. How would you solve this with coroutines?
task download-url
for url in urls:
download(url)
task extract-links
parsed = parse(document)
return parsed
task fetch-links
for link in document.query("a")
return link
task save-data
db.save(url, link)
How would you do control flow and scheduling and parallelism and async efficiently with this code?
* `db.save()`, `download()` are IO intensive whereas `document.query("a")` and `parse` is CPU intensive.
* I want to handle plurality or multiple items trivially such as multiple URLs and multiple links.
* I want to keep IO and CPU in flight at all times.
I think I want this schedule:
https://user-images.githubusercontent.com/1983701/254083968-b46485c8-fe5f-43ea-b840-d0d63dab4a51.PNG https://user-images.githubusercontent.com/1983701/254083968-...
I have a toy 1:M:L 1 scheduler thread:M kernel threads:N lightweight threads lightweight scheduler in C, Rust and Java
https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread
This lets me switch between tasks and preempt them from user space without assistance at descheduling time.
I have a simplistic async/await state machine thread pool in Java. My scheduling algorithm is very simple.
I want things like backpressure, circuit breakers, rate limiting, load shedding, rate adjustment, queuing.