31 ms·

Ask HN: What problem are you close to solving and how can we help?

• k1rcher 5y ago
  We are having atrocious READ/WRITE latency with our PG database (api layer is django rest framework). The table that is the issue consists of multiple JSON BLOB fields, with quite a bit of data— I am convinced these need to be abstracted to their own relational tables. Is this a sound solution? I believe it is the deserialization in these fields of large nested JSON BLOBS that is causing latency. Note: this database architecture was created by a contractor. There is no indexing or relations existing in current schema. Just a single “Videos” table with all metadata stored as Postgres JSON field type blobs. EDIT: rebuilding the schema from the ground up with 5-6GB of data in the production database (not much, but still at the production level) is a hard sell, but I think it is necessary as we will be scaling enormously very soon. When I say rebuild, I mean a proper relational table layout with indexing, fk’s, etc. EDIT2: to further comment on current table architecture, we have 3-4 other tables with minimal fields (3-4 Boolean/Char fields) that are relationally linked back the Videos table with a char field ‘video_id’, that is unique on the Videos table. Again, not a proper foreign key so no indexing.
  • taesu 5y ago
    imo, json dt should be an intermidiary step in db struc in rdb, never the final. Once you know & have stable columes, unravel the json into proper cols with indexing, it should improve the situation if youre having issues with 5gb, you will face exponential problems when it grows due to lack of indexing
    • k1rcher 5y ago
      Cheers for the response (and affirmation). After some latency profiling I am convinced proper cols with indexing will vastly improve our situation since the queries themselves are very simple.
      • taesu 5y ago
        You can take incremental approach and p.o.c with the data that you have so you can justify your move too!
      • esemathew 5y ago
        Depending on how much of the data in your json payload is required, extract data into their own table/cols. And store the full payload in a file system/cloud storage.
      • machiaweliczny 5y ago
        Also there's a way to profile which queries take longest via DB itself and then just run EXPLAIN ANALYZE to figure what's wrong
  • TekMol 5y ago
    I don't know about PG, but with MariaDB, a nice way to find bottlenecks is to run SHOW FULL PROCESSLIST in a loop and log the output. So you see which queries are actually taking up the most time on the production server. If you post those queries here, we can probably give tips on how to improve the situation.
    • k1rcher 5y ago
      Interesting. I believe I noted a similar function in the Postgres docs I was scouring through Friday. I’ll give it a look and see what I can find. Tangentially related for those who have experience, I am using Django-silk for latency profiling.
      • taesu 5y ago
        also, never trust orms. they make it easier to query but they do not use/output the most optimized quries
  • smackeyacky 5y ago
    Bit hard to tell without some idea of the structure of the data, but my experience has been storing blobs in the database is only a good idea if those objects are completely self contained i.e. entire files. If you write a small program to check the integrity of your blobs i.e. that the structure of the json didn't change over time, you may be able to infer a relational table schema that isolates those bits that really need to be blobs. Too leave it too long invites long term compatibility issues if somebody changes the structure of your json objects.
  • Rodeoclash 5y ago
    I had this issue at a previous job where we would query an API (AWS actually) and store the entire response payload. As we started out we would query into the JSONB fields using the JSON operators, however at some point we started to run into performance issues and ended up "lifting" the data we cared about to columns on the same record that stored the JSON.
  • sgk284 5y ago
    Are you just doing primary key lookups? If so, a new index won’t do much as Postgres already has you covered there. If you have any foreign key columns, add indexes on them. And if you’re doing any joins, make sure the criteria have indexes. Similarly, if you’re filtering on any of the nested JSON fields, index them directly. This alone may be sufficient for your perf problems. If it isn’t, then here’s some tips for the blobs. The JSON blobs are likely already being stored in TOAST storage, so moving them to a new table might help (e.g. if you’re blindly selecting all the columns on the table) but won’t do much if you actually need to return the JSON with every query. If you don’t need to index into the JSON, I’d consider storing them in a blob store (like S3). There are trade offs here, such as your API layer will need to read from multiple data sources, but you’ll get some nice scaling benefits here and your DB will just need to store a reference to the blob. If your JSON blobs have a schema that you control, deprecate the blobs and break them out into explicit tables with explicit types and a proper normalized schema. Once you’ve got a properly normalized schema, you can opt-in to denormalization as needed (leveraging triggers to invalidate and update them, if needed), but I’m betting you won’t need to do any denorm’ing if you have the correct indexes here. And since you have an API layer, ideally you’ve also already considered a caching layer in front of your DB calls, if you don’t have one yet.
    • k1rcher 5y ago
      This is super interesting stuff. First of all, I think the caching layer (which we currently don’t have) is going to be a necessity in the coming weeks as we scale for an additional project (that will be relying on this architecture) Second of all, it is just PK lookups. We don’t actually have a single fk (contractor did not set up any relations), which makes me think moving all of this replicated JSON data from fields to tables may help. The queries that are currently causing issues are not filtering out any data but returning entire records. In ORM terms, it is Video.objects.all(), and from a URL param in our GET to the api, limiting the amount of entries returned. What’s interesting is this latency scales linearly, and at the point we ask for ~50 records we hit the maximum raw memory alloc for PG (1GB) causing the entire app to crash. The solution you propose for s3 blob store is enormously fascinating. The one thing I’d mention is these JSON fields on the Video table have a defined schema that is replicated for each Video record (this is video/sensor metadata, including stuff like gps coords, temperature, and a lot more). So retrieving a Video record will retrieve those JSON fields, but not just the values: the entire nested BLOB. And does so for each and every record if we are fetching >1 Would defining this schema with something like Marshmallow/JSON-Schema be a good idea when you mention JSON schemas we control? As well as explicitly migrating those JSON fields to their own tables, replaced with an FK on the Video table?
  • Smaug123 5y ago
    I think your heart shouldn't quail at the thought of re-schemaing 5-6GB! I'm going to claim that the actual migration will be very quick.
    • k1rcher 5y ago
      This is an affirmation I’ve been longing to here, lol! I’ve already done the legwork, cloning to the current prod DB locally and playing around with migrations, but the fear of applying anything potentially-production breaking is scary to a dev who has never had to work on a “critical” production system!
      • vendiddy 5y ago
        I would recommend setting up a staging app with a copy of the production database, testing a migration script there, then running the same script on production once you're confident.
  • HarryTeunissen 5y ago
    Questions first: 1. What are the CRUD patterns to the "blobby data". 2. What are the read patterns and how much data needs to be read. Until Read/Write are properly understood the following solutions should be considered as general guide lines only. If staying in PG: JSON can be indexed in Postgres You could also support a hybrid JSON/Relational model giving the best of worlds. Read: Create views into the JSON schema that model your READ access patterns and expose them as IMMUTABLE Relational entities. (Clearly they should be as light weight as possible) Modify: You can split the JSON blobs into their own skinny tables. This should keep your current semantics and facilitate faster targeted updating. Big blobby resources such as video/audio should be managed as resources and not junk up your DB Warning: Abstracting the model into multiple tables may cause its own issues depending on how you ORM map your entities. Outside the Box Thinking: -Extract and transform the data for optimized reading. -Move to MongoDB or a Key Value store Conclusion: What are the update patterns? -is only 1 field being updated -inter-dependencies of the data being updated How are "update anomalies" minimized You will need to create a migration strategy to a more optimal solution and would do well to start abstracting with views. As the data model is improved this will be a continuous process and the data model can be "optimized" without disturbing the core infrastructure requiring rewrites.
  • 101008 5y ago
    I had to move a MongoDB to PG database on my new job (old contractor created MVP, I was hired to be the "CTO" of this new startup) and I had some problems at first, but after I created the related models and added indexes, everything worked fine. As someone said, indexes are the best to do lookups. Remember your DB engine do lookups internally even if you are not aware (joins, for example), so add indexes to join fields. Another thing that worked me for me (and I dont know if it's your case), was to add trigram text indexes, which make it faster to do a full text search. Remember, anyway, that adding a index makes search faster, but insert slower, so be careful if you are inserting a lot of data.
  • ufmace 5y ago
    I don't think the latency issues are necessarily related to the poor schema. I'd say to dig into the query planning for your current queries and figure out what's actually slow, since it may not be what you expect. Rearchitecting the schema might be worth doing. From the technical side, PG is pretty nice about doing transactional schema changes. I'd be more worried about the data though. Are you sure that every single row's Json columns have the keys and value types that you expect? Usually in this type of database, some records will be weird in unexpected ways. You'll need to find and account for them all before you can migrate over to a stricter schema. And do any of them have extra unexpected data?
  • mamcx 5y ago
    Other tips: - Change the field type from JSON to JSONB (better storage and the rest) https://www.postgresql.org/docs/13/datatype-json.html https://www.postgresql.org/docs/13/datatype-json.html -Learn the json in-build functions and see if one of them can replace one you made ad-hoc - Seriously, replace json with normal tables for the most common stuff. That alone will speed up things massively. Maybe keep the old json around in case, but remove when it become old(?) - Use views. Views allow to abstract over your databaee and allow to change internals - If a big things is searching and that searching is nkind of complex/flexible, add FTS with proper indexing to your json then use it as first filter layer: SELECT .. FROM table WHERE id IN (SELECT .. FROM search_table WHERE FTS_query) AND ...other filters This speeeeeedupppp beautifully! (I get sub-second queries!) - If your query do heavy calculations and your query planner show it, consider move them into a trigger and write the solved result into a table, and query that table instead. I need to loans calculations that requiere sub-second answers and this is how I solve it. And for your query planner investigation, this handy tool is great: https://tatiyants.com/pev/#/plans https://tatiyants.com/pev/#/plans
  • AdrianB1 5y ago
    Large blobs are not the use case of relational databases - this is the starting point for any such discussion. I have 2 current projects where I am convincing the app builders (external companies, industry-wide used apps) to change this, keep relational data in the database and take out the blobs, so far is going better than expected.
  • malux85 5y ago
    Examine the slow queries with the query planner, don’t spend a bunch of time re-architecting on a hunch until you know for sure why it’s slow! An hour with the query planner can save you days or weeks or wasted work!
• deleted 5y ago
  [deleted]
• Rodeoclash 5y ago
  I'm not sure if this is in the spirit of the thread but I've been working on a way to allow reviews of gameplay in video games. In short, you upload a video of you playing the game and someone who's an expert can review it. I currently have a UI with the comments down the side of the screen which looks like this: https://www.volt.school/videos/c980297a-417b-416f-947b-58a708d00efa https://www.volt.school/videos/c980297a-417b-416f-947b-58a70... This is good because you can easily: - See all the comments - Navigate between them - See replies etc. However it has a huge problem with you trying to balance watching the video with reading the comments. I also have an alternative UI I've been working on which only shows one comment at a time: https://www.volt.school/videos-v2/c980297a-417b-416f-947b-58a708d00efa https://www.volt.school/videos-v2/c980297a-417b-416f-947b-58... However the downsides of this is that you can't see all the comments at once. I'm not a UI/UX designer AT ALL so I'd really appreciate some pointers around how to think about making this better! The original post mentions "close to solving", I think I am pretty close but it's still not quite right and while I'm not out of ideas yet, I'd appreciate feedback if solving this is obvious to someone else.
  • DougBTX 5y ago
    So maybe the problem with showing all the comments at once is that there are too many, and when showing one at a time they are not shown for long enough. How about breaking the play into chapters/zones/rooms/segments (whatever makes sense for the game) then showing all the comments for that segment. Once the segment ends, there would be a replay button if they missed anything on the first play while reading comments, and a next segment button to carry on. Interesting time spans could be marked for slow motion, boring bits played at double time. There would be high level navigation between segments with thumbnails and comment counts. Buttons to skip between “pivotal moments”, maybe with voting to highlight them.
  • marshallbananas 5y ago
    How about showing the comments directly on the video, at a specific time, and a specific place. Something like Soundcloud comments but for video. Asian video platforms used to do that. Here's an example: https://www.youtube.com/watch?v=hOMMQmYwd4I https://www.youtube.com/watch?v=hOMMQmYwd4I It's totally crazy but can be made much more coherent. Would be useful to have comments at a specific time and place on the screen just for very accurate pointers/comments.
• the-dude 5y ago
  We are working on a totally new way to do cold fusion, our only problem is getting enough new fuel into the reactor without disturbing the running process. Any help would be greatly appreciated.
  • iamgopal 5y ago
    Disclaimer : I know nothing about anything. Hawking radiation ? Laser tunnel ? Magnetic canon ? Centrifugal launcher ? Vacuum diffusion ? Electrical beam lensing ?
    • desperate 5y ago
      Tesla valve?
    • a1369209993 5y ago
      > Hawking radiation ? That actually does make for a fairly efficient (better than fusion in energy per unit fuel mass) reactor design in principle, but you need a sub-solar-mass black hole (in the 1 billion - 100 billion ton range), and there's no known practical way to produce one.
  • mandmandam 5y ago
    Maybe the solution is in your fridge? ... Not being flippant; I find that these kind of prompts can help in thinking from a new approach.
  • Y_Y 5y ago
    What does your reactor look like? What fuel are you using? How are you getting to ignition?
  • minhmeoke 5y ago
    Could you use tiny pellets fed in by a linear actuator and gravity, rotary loader, or some kind of a conveyor belt? If you want an off-the-shelf solution that's easy to reload, perhaps you could repurpose the loading mechanism of a machine gun to dispense the pellets.
• cookiengineer 5y ago
  I'm working on a prototype that uses the compositional game theory [1] and adapts it to be able to reliably predict the order complexity of functors and their differences between states. A huge bonus there would be when the order difference can be represented in a graph, so that tesselation or other approaches like a hypercube representation can be used for quick estimations. (that's what I'm aiming for right now) If successful, the next step would be to integrate it into my web browser so that I can try out whether the equilibrium works as expected on some niche topics or forums. [1] https://arxiv.org/abs/1603.04641 https://arxiv.org/abs/1603.04641
• rambojazz 5y ago
  Who are "we"?
  • the-dude 5y ago
    I suppose it is just him and the driver.
  • mdp2021 5y ago
    Probably the HN participants ("Quickfire problems, quickfire suggestions")
• RHSman2 5y ago
  How to get my kids to bed to bed.
  • tmilard 5y ago
    Heu, raise your voice
  • xcambar 5y ago
    By no means am I an expert, but just another parent on HN. 1. Ritualize I notice a pattern with my kids: going to bed is a ritual, and any deviation is reason enough for them to leave the bed. 2. Slow down in advance Going to bed after playtime is impossible. So cut screens, playtime, play/listen some quiet music, read books or newspaper (again, no tablet/reader), at least 1h before bed time. 3. Recap the day Remind your kids of their day, activities and make them aware of the fatigue. Works better when in bed, with mine. 4. Stay with them if they're afraid Learn why they're afraid, teach them why there's no reason to be afraid. I've had to hang a sock to the door every night for months to scare tigers away :D It just works ^^ Every parent knows the pain and every kid has their own back story and the relationship with the parent(s) is key to finding a way into bed. Eventually, they will sleep. In our case, we settled that unless exceptional situations, our kids had to fall asleep in their own bed, because we wanted/needed our intimacy. To get that, I had to stay in my kids' room for as long as 2hrs for months, but didn't let go. Today, going to bed is thankfully not a situation anymore. I'm not sure this is in any way helpful, but here's my shared experience and learnings. YMMV.
  • 123pie123 5y ago
    in general aim for repetition and stringing activities together. So for us, when I first started to do this. Each night they get a 'treat' but to get that treat they first need to be ready for bed - eg bedroom ready to sleep in, correctly dressed/ washed etc.. then after the treat they must choose a calming activity - ideally in their bedroom eg reading (nothing that gets their heart rate up) for 30-60mins then they must bush their teeth It's this point we say time for bed, but we allow them to carry on reading for another 30-60min then it's lights off if they don't do the activities/ actions after the treat, then we warn them that they'll not get one tomorrow etc. (and really do what you say) also you may need to flexible on the activities until they get into the swing of it
• tmilard 5y ago
  https://free-visit.net https://free-visit.net : Like Matterport but with an FPS game engine. I am looking for my first client: Ideally someone in charge of a Museum/gallery or other grandiose indoor space.
  • TekMol 5y ago
    Hangs when I click "Try the example".
    • tmilard 5y ago
      Just Refresh the page : it will succesfully play (known bug)
      • TekMol 5y ago
        Now it works. I wish AirBnBs and hotel rooms would offer this type of preview of their premises.
        • deleted 5y ago
          [deleted]
  • mkl 5y ago
    Looks pretty slick. I am not a gamer, and the controls feel very backwards to me. You need to hold the mouse button to look in different directions, which makes it feel like you're dragging, but the direction you move it isn't the direction you're dragging the view. I don't think mouse capture is a good idea in the browser, and if you're aiming at museums and galleries, maybe reversing the mouse direction to make it like familiar dragging would be better. Edit: There's a bug where if you start dragging with the mouse and let go the mouse button outside the 3D view, it acts like the button is still held down (a bit like mouse capture) which was easier but quite confusing. It seems unusable with a touch screen on desktop. The fact that you can fly is useful but non-obvious. I ended up down at floor level and wondering how to see the pictures on the walls. The demo is in a /fr/ path, but is in English (Chrome offers to translate it to English, because it somehow thinks the English words are French), but then some parts of the interface like "Share your place" are in French.
• contingencies 5y ago
  Frustrated by the degree of manual programming process in production metal machining. The industry exists largely on inertia. I would like to resolve this by applying standard optimization algorithms to a set of known machining strategies plus machine, work-holding, material, part and tool inputs. Have already analyzed the problem space to some extent and will be touring a huge production facility next week to better understand best-in-class processes from large established players. Need someone to either wrap existing simulation algorithms (any CAM system) or write enough of one (not that hard, the solution space is extremely multivariate but well understood and well documented) to make it feasible (not too hard for 2.5D machining). You can get as intellectual as you like in the solution, but remember perfect is the enemy of done. Value is huge, happy to split equity on a new entity to resolve if a workable solution for the easier subset of parts emerges in the next few weeks.
  • iamgopal 5y ago
    We run about 40-50 CNC. Lots of our engineering time goes in to planning, on how to step by step machine a component so that it can reach mentioned tolerances. Sometimes required tolerance are at or below the machine accuracy. Are you going to solve this also ?
    • contingencies 5y ago
      I am looking at the low hanging fruit, 80/20 right now.
• NicoJuicy 5y ago
  I'm looking for the best way to implement Multitenancy, Submultitenancy Impersonation with JwtTokens and IdentityServer4 ( dot net). I'm curious on how other people solved it ( by cookies, subdomain, ... ) and if you used a JwtToken for it.
  • ignoramous 5y ago
    Looking for something like authzed.com?
    • NicoJuicy 5y ago
      I've got the database and query part covered. But I haven't decided yet on the actually flow. Where I'd identify the current tenant or impersonate him. + The influence of impersonation on that flow.
• yosito 5y ago
  This may already be solved, but one of the last pieces remaining in my quest to be Google-free is an interoperable way to sync map bookmarks (and routes, etc) between different open source mapping apps. I can manually import/export kmz files from Organic Maps and OsmAnd, and store them in a directory synced between different devices with Nextcloud, but there's no automatic way to keep them updated in the apps, and so far I haven't found a great desktop app for managing them either. The holy grail would be to also have them sync in the background to my Garmin Fenix, but I am not aware of a way to sync POIs to a Garmin watch in the background. Related: I'd love to have an Android app with a shortcut that allows me to quickly translate Google Maps links into coordinates, OSM links or other map links. There is a browser extension that does this on desktop, so if anyone is looking for a low hanging fruit idea for an Android app, this might be a fun idea (if I don't get around to it first).
  • tome 5y ago
    Have you documented your replacements for various Google technologies somewhere? I'm particularly interested in a good calendar.
    • yosito 5y ago
      I'm using Nextcloud to host my calendar. On my work Mac, I connect to it using Fantastical. On my personal Ubuntu machine I use GNOME Calendar, and on Android I use https://github.com/Etar-Group/Etar-Calendar https://github.com/Etar-Group/Etar-Calendar Everything is seamless for me, though admittedly I'm not a super heavy calendar user. I plan to do a write up on my whole Google-free setup, but I haven't done it yet, unfortunately.
      • tome 5y ago
        Thanks! Do you self-host Nextcloud or did you get an account at one of the providers?
        • yosito 5y ago
          I got a VPS and installed Nextcloud with Docker. I would self-host on my own server, but I'm too nomadic for that at the moment. I think the /e/ foundation has a decent managed Nextcloud setup.
• pedrokost 5y ago
  We are experiencing very high CPU load caused by tinc [0], which we use to ensure all communication between cloud VMs is encrypted. This is primarily affecting the highest traffic VMs, including the one hosting the master DB. I am starting to consider alternative tools such us wireguard to reduce load, but I am concerned of adding too much complexity. Tinc's mesh network makes setup and maintenance easy. The wireguard ecosystem seems to be growing very quickly, and it's possible to find tools that aim to simplify its deployment, but it's hard to see which of these tools are here to stay, and which will be replaced in a few months. What is the best practice, in 2021, to ensure all communication between cloud VMs (even in a private network) is encrypted? [0]https://www.tinc-vpn.org/ https://www.tinc-vpn.org/
  • ignoramous 5y ago
    NoCode: fly.io with its 6pn (out-of-the-box private networking among clusters in the same org). DIY: envoyproxy.io / HashiCorp Consul for app-space private networking over public interfaces. LowCode: Mesh P2P VPN network among your clusters with FOSS/SaaS like WireTrustee / tailscale.io / Slack Nebula.
  • PhilippGille 5y ago
    Apart from some smaller projects building on top of WireGuard, there's Tailscale [1]. One of the founders is Brad Fitzpatrick who worked on the Go team at Google before and built memcached and perkeep in the past. Outside of the WireGuard ecosystem there's ZeroTier [2] which has been around for a while and they're working on a new version; and Nebula [3] from Slack, which is likely to be maintained as long as Slack uses it. There might be others, but with tinc these four are the ones I've seen referred to most often. [1] https://tailscale.com https://tailscale.com [2] https://www.zerotier.com https://www.zerotier.com [3] https://github.com/slackhq/nebula https://github.com/slackhq/nebula
    • iyn 5y ago
      +1 for Tailscale, the product is great. I've used it in a very limited scale but can vouch for quality and performance. No CPU issues at all (even on rPi).
• yasserf 5y ago
  I’m facing an issue where I store small binary data blobs within a Postgres column in order to benefit from delete cascades. I’m considering moving the binary data into S3 and then doing the sync layer on the server (which means the front end requests the data from the backend and is given it back as a JSON object with base64 values). Doing this manually via code isn’t impossible, just API intensive, so I’m wondering if this is a solved issue for anyone. The why: The JSON blobs are recordings of words and sentences that can be copied between articles.
  • dafelst 5y ago
    It's hard to give any advice on this until you detail the problem you're trying to solve. For example: Where/why is your current system failing/inadequate/cumbersome? Why do you want to move the data to s3? Why are delete cascades important?
• cygned 5y ago
  I try to find an agile project management tool that works for us. We run on what many would call Scrum (it’s not actually Scrum). We are on JIRA now, and it’s … JIRA. We tried basically any other tool, including Excel (yes, that is somewhat possible). My problem generally is that tools are slow, planning is cumbersome, visibility is limited and reporting for clients is often even more limited. Heck, I’d even write my own tool if I knew it would help others, but I am concerned it’s too close to what we already have for anyone to actually migrate. You could help me by sharing your thoughts!
  • huksley 5y ago
    Depends on workflow and team size. For small teams good fit could be some kanban based tools, for example Trello or GitHub projects. You could also modern try agile tools, for example Linear. JIRA is good for 100+ teams and complex architectures.
  • wtf77 5y ago
    have you tried https://tara.ai/ https://tara.ai/?
  • MildlySerious 5y ago
    Is linear.app in the realm of what you're looking for? Have you tried that? Not affiliated, but I've had a positive experience with it in a small team. I would describe it as an IDE for issues.
  • 101008 5y ago
    Asana works really good for us. Really good UI/UX and fast, which I think it's the best feature they have :)
  • tomcooks 5y ago
    Try Asana
  • yardshop 5y ago
    I've recently started using ClickUp for managing my helpdesk and development work and I like it a lot. I don't do scrum myself but the product claims to be useful for that kind of work, as well as many other approaches and use cases. https://clickup.com https://clickup.com https://clickup.com/on-demand-demo https://clickup.com/on-demand-demo ClickUp for Agile Workflows https://www.youtube.com/watch?v=H9hZRwivnL8 https://www.youtube.com/watch?v=H9hZRwivnL8
• adammunich 5y ago
  I am working on two problems: 1. I want to create way to generate electrical power without pollution. Basically, a closed cycle process that releases no pollutants, or electronic waste. 2. I want to do everything I can to eliminate gender bias in the world.
  • znpy 5y ago
    Regarding 1, if you can accept some pollution at the beginning, hydro-electrical can be a solution, albeit probably not for a global scale. We have a small hydro-electrical plant una River near my house and really it's no big deal, it fits very nicely in the surrounding environment and it produces clean energy. It's also educational because since the river is near the city small children classes can visit it and learn about it.
    • ajot 5y ago
      Regarding hydro, you can go micro to power a house or some small comunity. There are lot's of books on microhydropower, but get a look at this fantastic post at ludens.cl http://ludens.cl/paradise/turbine/turbine.html http://ludens.cl/paradise/turbine/turbine.html
    • adammunich 5y ago
      I love hydroelectric power and I want the circuits I design for solar to be capable of utilizing the raw power from a small turbine as well without any hardware modifications.
    • minhmeoke 5y ago
      Rather than generating electricity directly, it might be more practical to reduce electricity consumption using other approaches: Geothermal can be a solution for generating electricity directly, but if you'd like to minimize electronic waste perhaps it would be easier to use it to replace alternative energy sources for HVAC purposes. Biofuels (eg: plant bamboo, grow it, then burn it) can also technically be closed cycle energy sources. Solar water heaters can also reduce electrical or fossil-fuel-based energy consumed for generating hot water.
  • 5y ago
• motyar 5y ago
  Problem: Create image with text. Solution: https://img.bruzu.com?a.t=Text https://img.bruzu.com?a.t=Text You can help by trying the API and giving honest feedback. https://bruzu.com https://bruzu.com
  • Rodeoclash 5y ago
    It seems kind of interesting, what are some use cases you've seen for it?
    • motyar 5y ago
      Use cases: 1. Image generation automation: Like into automation like posting tweets as image to instagram. 2. Image generation scaling: Create multiple images with just variable text, like greeting messages or product images or open graph images.
  • naveen_ 5y ago
    Check out: https://www.bannerbear.com https://www.bannerbear.com
  • mkl 5y ago
    The API docs' example URLs can't be copied and pasted, as "%20" gets inserted. I'm probably not the target audience for the API, but it's a neat idea. I think people will end up with images described with tons of magic numbers that relate to each other in invisible ways and become unmodifiable and unmaintainable. Variables might help, as might relative positioning and sizing of elements. Some feedback on the Designer: The size setting dropdown is quite strange. Choices of unfamiliar destinations don't seem to make sense, and some of the things that do seem familiar come out an unexpected size and shape (e.g. "infographic"). The pixel sizes are clearer, but better would be handles on the canvas that can be dragged. There's also a typo: "Choose form a list of sizes". Circles don't get resized? I can drag handles to make the apparent bounding box bigger, but the circle doesn't change: https://imgur.com/a/KhYluKj https://imgur.com/a/KhYluKj. Other shapes seem okay. Chrome 89 on Linux. The tutorial walkthrough pops up every time I go to the designer, even though I've been right through it.
• habibur 5y ago
  A search engine that prioritizes ad free, tracker free sites. Of course google can't do it. But this is a ripe for someone to step in.
  • fsflover 5y ago
    https://wiby.me/ https://wiby.me/
• davehcker 5y ago
  I have posted this here before- hexafarms.com. I am trying to use ML to discover optimal phenotype for growing plants in vertical indoor farms to a. have the higest quality produce b. to lower the cost of producing leafy green/med plants, etc. within cities itself. Basically, every leafy green (and herbs, and even mushrooms), can grow in a range of climatic condition (phenotype, roughly) ie temperature, humidity, water, CO2 level, pH, light (spectrum, duration and intensity) etc. As you might have seen around the world there is a rise in indoor vertical farms, but the truth is that 50% of those are not even profitable. My startup wants to discover the optimal parameters for each plant grown in our indoor vertical farm and eventually I would let our AI system control everything (something like alphaGo, but for growing plant X (lettuce, kale, chard, ). Think of it as reinforcement learning with live plants! I am betting on the fact that our startup will discover the 'plant recipes' and figure out the optimal parameters for the produce that we would grow. Then, the goal is that cities can grow food cheaper in more secure and sustainable way than our 'outsourced' approach in country side or far away lands. So now I have secured some funding to be able to start working on optimizations, but I realized that *hardware* startups are such a different kind of beast (I am a good software product dev though, I think). Honestly, if anyone with experience in hardware related startups (or experience in the kind of venture I am in) would just want to meet me and advise me, I would take it any day. Being the star of the show, it's hard for me to handle market segmentation, tech dev, team, next round of funding, European tech landscape, etc. I am foreseeing so many ways that our decisions can kill my startup, all I need is advise from someone qualified/experienced enough. My email: david[at]hexafarms.com
  • mattmanser 5y ago
    I'm not able to help, but you don't have any contact details listed against your profile or in this post. How is anyone able to contact you? At the very least what's a link to your startup's website?
    • davehcker 5y ago
      Sorry I thought my email was on my HN profile. I am sitting behind david[at]hexafarms.com
• bsldld 5y ago
  I am not there yet, but I am trying to make education loan-free and based on equity. The details of the project are written here: https://loan-free-ed.neocities.org https://loan-free-ed.neocities.org
  • dane-pgp 5y ago
    > when that student starts generating income then a small percentage from that income is auto-deducted and distributed to everyone who was involved in that student's education. Your idea, if implemented well, may end up being a net positive for society, but I can't help imagining a future where every child, from the moment they are born, has a biometric ID connecting them to a consortium of companies which provide their education, health care, housing, energy, internet connectivity, transport, media access, and so on. It would be like living in a company town, being paid in company scrip, except you wouldn't notice the restrictions (as long as you kept earning). If you ever increased your income, your consortium might let you choose whether you want to upgrade your housing or your health care plan, but if you lost your job, they'd force you to take one of their choosing and downgrade your plans if it had a lower salary. In this dystopia, all consumables from food to toilet paper would presumably be sold by Amazon, and other items like furniture and electronics would be provided as a service so that you rent them from your consortium. The only question is why people wouldn't try to undo this system through the political process, but then we might ask that about the current system.
    • bsldld 5y ago
      > Your idea, if implemented well, may end up being a net positive for society Thank you! > but I can't help imagining a future where every child, from the moment they are born, has a biometric ID connecting them to a consortium of companies which provide their education, health care, housing, energy, internet connectivity, transport, media access, and so on. My idea will not have that side-effect because not only is the project non-profit and open source, but it is also decentralised. And if we keep thinkng of a dystopian future then we won't be able to do anything positive unless we become some sort of social revolutionaries. I don't have those skills. But I can think of small ideas to make a positive impact that benefits everyone though. The idea I listed in my original post above is very simple, it helps teachers, lecturers, or anyone who contibutes to a persons monetary decent life via education get appropriately paid for their efforts, and everyone(i.e. businesses) who benefits from an educated person should contribute towards that. It is a simple idea but notoriously diffcult to deploy because there is a possibility of this getting caught up in a political slugfest.
• stavros 5y ago
  I built a community that aims to keep FOSS projects alive. It's meant to solve the kitchen and egg problem by having as many people and projects sign up, and then any developer who was interested could just automatically get commit permissions to any project. It's called Code Shelter: https://www.codeshelter.co/ https://www.codeshelter.co/ It's stalled for a while, so I don't know how viable it is, but I'd appreciate any help.
  • smallerfish 5y ago
    One thing you could try to solve is coordinated revival of abandoned projects - i.e. extending your model to support unsolicited takeover of projects, in the case of a maintainer's having walked away. For example, I use a javascript library that's best in class for what it does, and yet hasn't had any real commits from its maintainer since 2016. There are 50 pull requests open, some of which fix significant bugs, or add good new features. There are literally 2000 forks of the library, some of which are published on npm but are themselves unmaintained, and almost none of which link back to the actual fork's code from npm. It's a mess, and I bet it's a situation repeated hundreds of times over. If you were to figure out a workflow by which a maintenance team could form on your platform, and then a) the existing maintainer is pinged to request that they add the team, falling back to b) making it easy for the new team to fork and adopt existing pull requests while supporting them through initial team-forming by laying out a workflow for assigning needed-roles, then I think you'd have a valuable platform. The key thing is ensuring there's a large enough team to start, so that yet another fork doesn't die on the vine, so maybe think about a(n old) reddit link type interface where people can link to, vote on, and volunteer for projects, with no work needed until there's critical mass and the platform moves the project forward.
    • stavros 5y ago
      Hmm, that's an interesting idea, thanks. Given that finding one maintainer is already hard, though, I think finding a team would be almost impossible...
      • smallerfish 5y ago
• huksley 5y ago
  How visually implement process in the app? I.e. how to guide users over complex process they need to do in the app to achieve success? The process might span different medium (write email, do something in the app, check twitter, etc) and different activities multiple days. How to make sure they know what they should do next? Checklist? Emails? Slack? Wizard?
  • tompazourek 5y ago
    Interesting... My to-go solution for this would be a detailed wiki page with screenshots, and link to that from a bunch of places. But I guess that's not an ideal solution really.
    • cuchoi 5y ago
      What do you use for the wiki?
  • gverrilla 5y ago
    World of Warcraft
  • janderson215 5y ago
    This would probably require a lot of front loaded work in your case, but if you need to train a boat load of people with very few (or zero) trainers, my favorite way to do it is (Atlassian’s Atlaskit Onboarding/Spotlight components)[https://atlaskit.atlassian.com/packages/design-system/onboarding https://atlaskit.atlassian.com/packages/design-system/onboar...]
  • le-hu 5y ago
    we use little floating checklist from userpilot.com
• inglor_cz 5y ago
  I am a bit ashamed to ask about such a trivial topic on HN, but I am not really sure how AJAX in Wordpress plugins works. I have a plugin that exports some WooCommerce orders into XLS. I would like to add a progress bar via AJAX, because the export may take very long for thousands of orders. But I am not really sure how to use AJAX in context of Wordpress specifically. I would love to see a minimal functional example, a simple plugin that does something similar. So far, all the plugins I saw were pretty convoluted and I lost my track around the code. (On a related note: a library of elementary examples for Wordpress plugin development would be nice. Like "This is how you create a menu entry.")
  • janderson215 5y ago
    In my experience, accurately reflecting the progress of and AJAX request, so I’ve seen a lot of people (myself included) take the lazy way out and just show an indeterminate spinner or bar just to show that stuff is happening.
    • inglor_cz 5y ago
      The idea entered my mind, but I am not happy with such cop-out, especially on my own site ... I would at the very last like to see iteration progress, which, while not 1:1 with time, is at least informative.
    • tsian2 5y ago
      It takes a bit of code but if you know the length of the response you're expecting then you can use XHR's "progress" event. Just be aware that the event will happen frequently and contain all the data so far, to avoid inefficient parsing and substring related memory leaks. I think his problem might be more about using JS in WordPress though.
  • 101008 5y ago
    First of all, you'll need to make PHP to display "progress", probably you'll need to override ob_start() or something like that, and find a format that let you append new progress on the response on the fly. I guess you already have an URL on your Wordpress setup that triggers this export. Let's call it {url}/export. Wordpress already has jQuery by default included. So'll you need to call that URL using jQuery $.post and then, accordingly to the response, update your progress bar. There is nothing specifically about Wordpress on this, besides the fact that you need to setup your own URL on Wordpress to do this, and then include your own JS after jQuery. That's all. If you find this too-complicated, a quick-hack is to create a page on WP Admin called Export Tool, and then on your theme create page-export-tool.php. That .php will be called when visited that Export Tool page.
• yosito 5y ago
  Ok, I'm double dipping. Another problem I'm trying to solve is: I've got a database of several hundred interesting conversation questions that I've collected over the years. Essentially just strings, though I've attempted to categorize them, rank them, and add other metadata. I'd like to figure out a way to sort them or dedupe them based on semantic similarity, but I'm not sure how to determine semantic similarity without painstakingly going through and manually looking for similar questions. Any suggestions on how to solve this would be welcome.
  • fshbbdssbbgdd 5y ago
    Put the questions in some semantic embedding space. Now you’ll have a vector representing each question. Then for each question, you can sort all the questions by how far the Euclidean distance is between their vectors. Or use some clustering algorithm like k-means to find clusters. By web search I found this to tutorial to put sentences in an embedding space: https://github.com/BramVanroy/bert-for-inference/blob/master/introduction-to-bert.ipynb https://github.com/BramVanroy/bert-for-inference/blob/master... I did not read this and am not endorsing it, but it looks like it’s doing roughly what I’m suggesting.
    • johnmoberg 5y ago
      Yep, exactly this. Check out sentence-transformers https://pypi.org/project/sentence-transformers/0.3.0/ https://pypi.org/project/sentence-transformers/0.3.0/, they have some great pre-trained models. Once you have the embeddings you can just compute the cosine similarity.
  • phenkdo 5y ago
    You could use BERT or Word2Vec or GLOVE. They are very simple to use with HuggingFace's library.
• mathgladiator 5y ago
  I've been thinking about two interesting problems. First, differentiating code to build a client side predictor with privacy as a consideration. I have code that describes how to translate domain messages into state changes, and I'm trying to figure out how to predict the effects of sending a message on a client even though the client has imperfect knowledge. Second, AI for games with imperfect information. Specifically, how to build an AI for Battlestar Galactica. These are in context of http://www.adama-lang.org/ http://www.adama-lang.org/
• michaelbuckbee 5y ago
  I could use some help with some heuristics for Machine Learning, like how much data do I need to make a workable model, what framework/approach makes more sense given my ultimate goals. Here's an example: there's a lot of ML tutorials on doing image identification. Like you have a series of images: picture one might have an apple and a pear in it, picture 2 might have an apple, orange, and a banana in it. Where I'm struggling is putting this into my domain. I have a 100k images and from that around 1k distinct labels (individual images can have between 1 to 7 of these different labels), with between 13,000 to 100 images as examples in each label. Is that enough data? Should I just start working on gathering more? Is this a bad fit for a ML solution?
  • ad404b8a372f2b9 5y ago
    Computer Vision is the one domain where ML (and neural networks in particular) is the undisputed king. Unless you're in an embedded application where you can't use neural networks in which case you might want to go with handcrafted features to trade-off accuracy for speed/compute-efficiency. With regards to the size of your dataset, there's no hard rules it depends on the complexity of the task. Problems with a high number of classes are among the more difficult ones, 100 samples per class might be enough or it might not . The only way to know for sure it to try and see if you reach a performance that's acceptable for your application. I recommend the Pytorch framework, it's coherent, easy to use and well documented (both the API reference and the examples available on SO and throughout the net). Your problem is similar to imagenet (assuming you want to detect the presence of a class and not its position, in which case it's a different problem.) so you can try to run one of the pytorch tutorials and see how well it does. The only difference is you want to detect multiple classes in one single image so you'll have to adjust your output layer and loss potentially but the network itself could remain the same. You also might want to look into doing transfer learning with a pretrained imagenet network to speed-up the training.
  • Y_Y 5y ago
    The answer is, "it depends". My recommendation would be to just start trying. What you describe sounds like you could train it in a couple of days. Grab an off-the-shelf resnet and just see what happens. This is a well-studied problem, you can just look at papers that train on imagenet, and then tweak their approaches.
• vcklnbagoil 5y ago
  Working on trying trying to extract the most benefit from my suicide while reducing the risk of of damage. Right now there's two issues: * I'd like to donate my kidneys, liver, and lungs as a living donor. However I've heard that there may be a psychological screening component for being a living donor which may detect that I'm planing on killing myself and trigger intervention. Is there a way around it? It's not critical for success but it would be nice to give back to someone else a chance at life, but I'm not sure if it's doable. * I want to minimize the possibility of being discovered and identified after death. I've selected a heavily wooded site that's time consuming to access on foot and selected timing to be early autumn some years from now once I've acclimated my acquaintances to my absence. The concern however comes from the fact that I have a security clearance and my fingerprints are in the system. I'm trying to figure out if it's a real concern that my body is discovered with fingers intact, and what needs to be done to mitigate it. Or if seclusion, exposure, and scavengers are enough to cover that possibility and it's not a real concern.
• OnlyRepliesToBS 5y ago
  can entropy be reversed?
• pollomarzo 5y ago
  I'm looking for a way to integrate a React app with an existing Vue... thing. Don't really need any communication between the two, just displaying it would be fine. My issue is: the Vue code just throws in <script> tags in the html and expects global variables (location instead of window.location), while the React code uses ES6 imports. The only partly working way I found is including <script> tags with a useEffect, but that doesn't play nicely with apex-charts for some reason, and includes forcing the html with a dangerouslySetInnerHTML after importing the existing file as a long string. Sub par, obviously. In addition, I'll probably need to include different vue apps in a couple different instances. Any suggestions? I think I might just keep them separate and open a new tab for the explanatory session. thanks! Reasoning: helping with the code behind a paper on explanatory AI systems. Related code on github if curious https://github.com/pollomarzo/map-generation/tree/main/graph https://github.com/pollomarzo/map-generation/tree/main/graph EDIT: thanks for suggestions will give them a spin throughout tomorrow :)
  • mindri0t 5y ago
    Have you tried SingleSPA? I’ve used it in the past to get React apps inside an old AngularJS app to replace parts of it over time and it worked pretty well. Docs say it works with Vue as well but I don’t have any direct experience with Vue at all far less for this kind of task so I’m not really sure it will work but it is worth looking at https://single-spa.js.org/ https://single-spa.js.org/
  • janderson215 5y ago
    Have you tried using `useRef` and attaching the Vue component to a parent React node? I haven’t does this with Vue, but have with a vanilla JS chart library.
  • tyingq 5y ago
    >just displaying it would be fine Iframes with postmessage() where needed (like dynamic window size changes) isn't pretty, but it's easy to do.
• jordwest 5y ago
  I want to bring back old school distributed forum communities but modernise them in a way that respects attention and isn’t a notification factory. Mastodon is a pretty inspirational project but the Twitter influence shows, I miss the long form writing that was encouraged before our attention spans were eroded. Not at all close to solving it, but it’s been on my mind for a long time. Would love to hear if there are others like me out there and what you imagine such a community to look like.
  • ted0_2021 5y ago
    I'm very interested in this space too. I want to start a project that explores different ways to communicate on the web. The current state of Facebook and Twitter are not the best way in my opinion.
    • codingclaws 5y ago
      You may be interested in my open source forum: https://github.com/ferg1e/peaches-n-stink https://github.com/ferg1e/peaches-n-stink It's basically an experimental communication platform. Right now I am building Internet forum style communication but I want to expand to other communication mechanisms later.
  • buro9 5y ago
    I tried this, got quite far. Will go a little further when I have spare cycles for it. Example: https://www.lfgss.com https://www.lfgss.com based on code you'll find on GitHub under microcosm-cc
  • 6510 5y ago
    I see a great blog one time where the author was writing a book in the first post. All following postings were dressed up change logs with many interesting and useful comments. You don't want the notification factory but if its a single post that gets updated you don't get any feed updates at all. Reading the same book again and again looking for the updated sections is also not much fun. Dropping a comment in the long list of comments under it doesn't really create a discussion. (specially not without feed updates) You could design the publishing tools so that it "forces" the user into that pattern. People could work on multiple books or long reads but start with a crappy draft or just a bunch of links.
• jeanlucas 5y ago
  I have an interesting challenge: I want to add IoT tools in several apartments and allow my guests to control them through an app. Currently I already have an app, but no IoT integrations (basically just reservation). How to do it safely? Also, which automations would be nice to have but not so obvious? Thinking from things like light control, but also checking the apartment energy consumption in order to detect appliances that need maintenance before they break.
• alecmgo 5y ago
  I want to make technical recruiters better at their job. Many sourcers and recruiters don't have a technical background and find it very difficult to hire software engineers, especially in the current labor market which is very tight. I'm starting off simple: writing recruiting guides from a software engineer's perspective that are easy to understand. Are there other ways we can make technical recruiters better?
  • whoisjohnkid 5y ago
    > Are there other ways we can make technical recruiters better? - list salary range for positions - emphasize tech stack - emphasize number of rounds I’ve wasted my time in the past going through the interview process only to find out the company’s budget for the position was only up to X after getting the offer; a valuable lesson I’ve learned to avoid since then of course. I also see some recruiters only talking about what the business does … leaving out the tech stack. If these points are clear and easily visible to recruiting leads they might get higher quality candidates. Just my two cents. ¯\_(ツ)_/¯
  • edmundsauto 5y ago
    I’ve also been thinking about this problem space. My approach is to help candidates build skills, demonstrate proficiency, in a loop with the recruiters. Basically, take the whole “how I learned my data science skills” into something that can be done in public. The recruiters can then see a wide range of examples, and can be better at picking up where people’s strengths and talents are. (This is focused on analytics)
• evandwight 5y ago
  I'm trying to make social media moderation more democratic and using that to decide fuzzy questions like "should this post be censored", or "is this misleading" [0]. While the crowd's answer won't be perfect it will help sort through a lot of the noise and feel better than the decision of whatever mod happened to create the subreddit. The problem: how can I make decisions based on a sample with a binary question. I think the central limit theorem applies, and I need to account for various priors and missing votes. Is there an existing solution to this problem? The server is written in nodejs, if that matters. [0] - https://efficientdemocracy.com/about/what-is-this https://efficientdemocracy.com/about/what-is-this
  • chriswarbo 5y ago
    This might be a good use-case for the "bayesian truth serum" http://economics.mit.edu/files/1966 http://economics.mit.edu/files/1966 This applies when our questions are not just trying to learn about the world (e.g. 'Our survey discovered that 10% of posts are considered misleading'); we are going to use their answers to decide on actions, e.g. removing posts, attaching warning labels, etc. Those answering the questions know this, and (if they have a preference over which action will be taken) are incentivised to give more extreme answers. A classic example is an ice cream company surveying shoppers about the flavours they like: if I truthfully answer that I like chocolate slightly more than strawberry, this will have a small effect on the survey result, and hence the company's new product flavours. However, if I falsely say that chocolate is the best flavour I've ever encountered, and that strawberry makes me vomit, that will have a much stronger effect on the survey result, and make it more likely that the company will make the chocolate ice cream that I prefer. The "bayesian truth serum" counteracts this by asking each question in two parts: there's the initial question we want answered, as well as an additional question: "how do you think others will answer?". For example: - "I find this misleading" and "I think 80% of respondents will find this misleading" - "I rate strawberry as 4/5" and "I think 10% of respondents will give strawberry 1/5; 20% 2/5; 50% 3/5; 15% 4/5; and 5% 5/5" The first answers (the ones we care about) are weighted based on two conditions: how closely the estimated distribution matched the real answers, and how 'surprisingly popular' the first answer is. To see why this cancels-out the incentives to lie: our best chance of affecting the result is to choose a 'surprisingly popular' answer, since this will contribute more weight to the result. However, these two constraints exactly cancel out: - The answers we predict are popular, will also be those we predict are unsurprising (after all, we could predict them!) - The answers we predict will be surprising, will also be those we predict are unpopular (that's why it would be surprising if they were popular!) It turns out that the rational strategy, for swaying decisions as much as possible towards the outcomes we want, is to answer the first part truthfully. A similar analysis applies to answering the second question (the estimates) truthfully. In that case there are two things to consider: - We want our estimates to be as close as possible to the true distribution, in order to maximise our response's weight. - We want to engineer our estimates such that the answers we disagree with get a high estimate, and hence appear 'unsurprising' (reducing the weight of those responses). Our estimates must sum to 100%, so decreasing the 'surprisingness' of one answer must increase the 'surprisingness' of the others. The effect we have on each answer's weight will be small, but it will affect every response which chooses that answer. Hence to have the largest impact, we need to decrease the 'surprisingness' of those answers we think will get the most responses. Yet that exactly what we've been asked for (an estimate of how popular we think each answer will be!)
• ooaaythrwwy 5y ago
  This isn't a technical question, but it's a problem I'm going to have to solve soon! I just accepted an EM role at a FAANG. This is a career "boomerang" for me - I was an engineer in the past, but then moved into technical support management. I'm coming back to engineering, but this is the first time I will have done it at a large company. All of my engineering experience has been at small scrappy startups where we just did everything, did it fast, and prioritized by whatever was most on fire. I don't think I've ever actually done a proper "sprint". While I have written a lot of code, the workstyle on my new team will be almost entirely foreign to me. Whose got pro tips for leading an engineering team in a large organization? What makes a high-powered team? What are the easy mistakes that will drive us into the ground?
  • bestcoder69 5y ago
    (Caveat: not an EM myself) This guy made the switch from dev to EM and has written articles on running effective teams,(e.g. one on how to help devs act as project leads) and even has resources like the docs he sends to tech leads when they start a new project, w/ lists of responsibilities etc: https://blog.pragmaticengineer.com/things-ive-learned-transitioning-from-engineer-to-engineering-manager/ https://blog.pragmaticengineer.com/things-ive-learned-transi... (just linking to the most directly applicable article but definitely browse around)
  • eatonphil 5y ago
    I've been an EM at a startup and an F500. Don't be scared to have less meetings than your peers. I basically only do one 1-hour sprint meeting each sprint to review the last sprint and to plan for the next one. It may help to have standups daily at first but you can often make them less frequent over time. Also, always avoid meetings to ideate. These are some of the most common and they are a huge waste of time compared to listing out ideas in a doc and having people review that asynchronously. And yet these meetings have a tendency to get called all the time. For example "there was a fire, let's get all the leads/EMs/directors together for 1 hour to figure out how to avoid this next time". Your two most important duties are 1) making sure your developers are given space to implement what's important and 2) building relationships throughout the company to better anticipate future needs that helps you do #1. Happy to chat anytime as well, email in profile.
• akkartik 5y ago
  Short version: I'm close to figuring out how to encourage more prototyping of software by making tests super easy to write during the prototyping process, and so de-risking rewrites. But one problem I've been stymied by is how to represent expectations of screens when they contain graphics. Long version: My Mu project (https://github.com/akkartik/mu https://github.com/akkartik/mu) is building a computing stack up from machine code. The goal is partly to figure out why people don't do real prototyping more often. We all know we should throw the first one away (https://wiki.c2.com/?PlanToThrowOneAway https://wiki.c2.com/?PlanToThrowOneAway), but we rarely do so. The hypothesis is that we'd be better about throwing the first one away if rewriting was less risky. By the time the prototyping phase ends, a prototype often tacitly encodes lots of information that is risky to rewrite. To falsify this hypothesis, I want to make it super easy to turn any manual run into a reproducible automated test. If all the tacit knowledge had tests (stuff you naturally did as you built features), rewriting would become a non-risky activity, even if it still requires some effort.Turning manual tests into automated ones requires carefully tracking dependencies and outlawing side-effects. For example, in Mu functions that modify the screen always take a screen object. That way I can start out with a manual test on the real screen, and easily swap in a fake screen to automate the test. Hence my problem: How do you represent screens in a test? Currently I represent screens as 2D arrays of characters. That is doable a lot of the time, but complicates many scenarios: * Text mode character attributes. If I want to check the foreground or background color, I currently use primitives like `check-screen-in-bg` which ignores spaces in a 2D array, but checks that non-spaces match the given background attribute. In practice this leads frequently to tests that first check the character content on a screen and then perform more passes to check colors and other attributes. * Non-text. Checking pixels scales poorly, either line at a time or pixel at a time. A good test should seem self-evident based on the name, but drawing ASCII art where each character is a pixel results in really long lines or stanzas. So far I maintain separate buffers for text vs pixels, so that at least text continues to test easily. * Proportional fonts. Treating the screen as a grid of characters works only when each character is the same width. If widths differ I end up having to go back to treating characters as collections of pixels. So Mu currently doesn't support arbitrary proportional fonts. * Unicode. Mu currently uses a single font, GNU Unifont (http://unifoundry.com/unifont/index.html http://unifoundry.com/unifont/index.html). Unifont is mostly fixed-width, but lots of graphemes (e.g. Chinese, Japanese, Korean, Indian) require double-width to render. That takes us back to the problems of proportional fonts. Currently I permit just variable width in multiples of some underlying grid resolution, but it feels hacky. Can people think of solutions to any of these bullets in a text-based language? Or a more powerful non-text representation?
• erikerikson 5y ago
  How do we scale social accountability and knowing? This is expected to enable us to solve distributed coordination problems. Also, it should facilitate richer more meaningful relationships between people. Expected outcomes include increased thriving and economic productivity. [edit: consider the limit on how many people you can know and the relationship between how deeply you come into relationship with that population and the size of that number]
  • dzink 5y ago
    The question is too far up in fuzzy space. Narrow down to several use cases and specific problems within those and search field will be more manageable. Examples: Social workers want to be able to handle more cases appropriately. How many cases can they handle without diminishing quality scores. Politicians want to appear caring to the needs of as many constituents as possible. How do they group needs into buckets to find what is most relevant. Find the overlap and dig into it with more cases and then questions.
    • erikerikson 5y ago
      Like automating the analysis of a recorded argument according to Gottman institute and other social heuristics for augmentation of marriage counseling services? [edit: i.e. count positive and negative sentiment statements assigned to speakers and compare the per speaker ratio to the experimentally determined minimum "healthy" ratios not yet replicated] You're right that there needs to be a tractable starting place. This is not lost on me. I may have used a flexible definition of "close to solving" but one's interpretation also fits into the scope of the effort. I'm at least 10% into it! ;P
  • slx26 5y ago
    I have spent quite a lot of time thinking about coordination in general. Indeed, knowledge is a vital part of it. The problem that I see is that knowledge is too vague and lossy and changing and incomplete [as I mentioned in this comment https://news.ycombinator.com/item?id=26203718 https://news.ycombinator.com/item?id=26203718]. An hypothetical solution would be a system that talked a language similar to plain english, but that was determinist. You let people write their problems and views to the system, and the system determines which are the widest consensus available within a given scope and what are the highest priority problems (perceived by people). This has a lot of problems, but it's a good way to think about the topic. Even with such a system, would you really be solving the problems you want to solve? If it does, then this is basically symbolic AI. You can try to relax requirements... but you kinda need an "automatic coordinator". If you go with a manual coordinator instead, then I doubt you will be able to scale anything that's not extremely rigid and hierarchical, at which point you are re-introducing many of the same problems you were trying to fight in the first place.
• chubot 5y ago
  These are statistics/math problems that 2 medical professionals I'm seeing are working on, not my own work. But they got me curious. FWIW I worked in "data science" as a software engineer for many years, and did some machine learning, so I have some adjacent experience, but I could use help conceptualizing the problems. Does anyone know of any books or surveys about statistics and medicine, or specifically mechanics of the human body? - One therapist is taking measurements of say your arm motion and making inferences about the motion of other muscles. He does is very intuitively but wants to encode the knowledge in software. - The other one has an oral appliance that has a few parameters that need to be adjusted for different geometries of the mouth and airway. The problems aren't that well posed which is why I'm looking for pointers to related materials, rather than specific solutions (although any ideas are welcome). I appreciate replies here, and my e-mail is in my profile. I asked a former colleague with a Ph.D. and biostats and he didn't really know. Although I guess biostats is often specifically related to genetics? Or epidemiology? I guess the other thing this makes me think of is software for Invisalign or Lasik, etc. In fact I remember a former co-worker 10 years ago had actually worked on Lasik control software. If anyone has pointers to knowledge in these areas I'm interested.
  • ad404b8a372f2b9 5y ago
    I can't help you conceptualize these specific problems but having worked on similar problems in the past I'd advise you to look into ordinary differential equations applied to those systems. They're used a lot for modelling in medical science and even if you're not interested in the dynamics of it they might lead you to the relevant literature for your problems and will address the parameters you're interested in and how they relate to each others.
  • jtolmar 5y ago
    > One therapist is taking measurements of say your arm motion and making inferences about the motion of other muscles. This seems like a sequential Bayesian filtering problem. Probably high enough dimension that you should just use a particle filter. The big seminal background text in this area is Bishop: Pattern Recognition and Machine Learning. If the "motion of other muscles" is inferring pose, you could also look into what computer graphics calls inverse kinematics (a typical IK model has a number of dimensions that could fit into a particle filter). There's some more in-depth stuff in motion planning that actually takes into account muscle capability. But I wouldn't know where to find info on that, short of watching the last several years of Siggraph Technical Papers Trailers, grabbing all the motion planning ones, then reading everything they cite.
• franze 5y ago
  https://share.securrr.app https://share.securrr.app A secure client side encrypted document (Passport / Id) sharing service Needs a DSVGO lawyer as part of the team.
  • 101008 5y ago
    I like this. I dont know if it is your goal, but I'd like to see you (or an alterantive) to succeed and be used by big companies, a la Stripe. Need to share a document? No need to reupload, it's already part of Securr. Share with the company for a limited period of time, Securr make it hard to copy and to store (and illegal), etc.
  • ta988 5y ago
    I love that idea, you should submit that on frontpage.
    • franze 5y ago
      Can't. As the legal framework does not exist yet. That's why I need a lawyer in the team.
      • DoreenMichele 5y ago
        You could potentially submit it as a Show HN and make it clear you very much need a lawyer to take the next step.* Your call, of course. I'm just a random internet stranger spitballing ideas here for how to take that next step and I know next to nothing about this problem space. Best of luck, whatever you choose to do. * Make sure you read the rules for Show HN and use your own judgement there as to whether this qualifies. https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
    • tomcooks 5y ago
      Frontpage?
      • ta988 5y ago
• ibaikov 5y ago
  I could really use some help with adversarial attacks. If there's someone trying to use CV to recognize stuff and we are trying to prevent it (basically, a black box situation) - is it viable to use adversarial attacks at all? Will it work long term? Can they overcome our AA by downsampling and adding noise? Can we make another AA that would still prevent them using CV? If it's better to chat elsewhere - my Twitter is in my profile.
  • high_byte 5y ago
    every week I read about autonomous driving adversarial attack, so I would say it is beyond feasible.
  • ad404b8a372f2b9 5y ago
    If you have access to their model it's possible, and adversarial attacks can be made to survive different sorts of processing. But like all piracy prevention tools it's a game of cat and mouse so if you're working against an intelligent adversary it'll come down to how many resources each of you can contribute to this fight. (Btw your twitter is not in your profile. Not that I have anything to add in private, just thought you should know if you're relying on it.)
    • ibaikov 5y ago
      No, the case is not having their model. I understand it's a game of cat and mouse, but this one case is gonna have a very quick iteration cycle, so I'm not sure if it's worth trying at all, it's not the same thing to test an adversarial attack on a self driving car once. Thanks, it didn't update my profile for some reason, fixed!
• myWindoonn 5y ago
  I'm working on destroying proof-of-work blockchains. I have a plan for BTC, but I'm not sure how to approach ETH. Advice would be appreciated.
  • high_byte 5y ago
    for eth - just wait?
  • betwixthewires 5y ago
    Interesting, what approach are you taking with BTC and what's the reason that approach won't work for ETH?
    • myWindoonn 5y ago
      I imagine overwhelming miners with legitimate but expensive work. The way to generate this work is relatively sensitive to low-level blockchain parameters. BTC happens to fit the approach, but ETH doesn't. This isn't surprising because the approach was originally invented for BTC alone, prior to the proliferation of many different blockchains.
  • jlokier 5y ago
    If you're interested in eliminating proof-of-work for ETH, you should really take a look at the proof-of-stake network in progress. Keywords: "Eth2", "proof of stake" and "the merge". When proof-of-stake takes over, there won't be any miners. The block proposal process is done by stakers instead. Some of the incentive issues with miners still exist with stakers, but raw competitive power consumption isn't one of them. It's true that proof-of-stake has been talked about for years, but it has picked up momentum since late last year, as the staking network was actually launched. The proof-of-stake network has been staking real ETH since end of last year, but does not yet handle mainnet ETH contract transactions. It's called Eth2, but that's caused some confusion, because it's not really a second version to run alongside the first, it is the R&D branch into proof-of-stake and other technical improvements, with mainnet ETH expected to adopt it in due course. So, the Eth1 components have been renamed "execution layer", Eth2 components renamed "consensus layer", and through a series of testnets and API developments which have been quite active this year, a big change called "The Merge" is being worked on by multiple funded groups (for client diversity) of core Eth developers at the moment. The Eth2 staking network that already exists has demonstrated the viability, and the investment of real ETH in serious quantities has built up some cryptoeconomic stability prior to its deployment as the ETH consensus layer. The time lag is intentional - you don't want to suddenly switch all ETH over to a network with too few invested stakers.
• dbancajas 5y ago
  How to find motivation/energy to do a long-term creative project when having a full time jobs + other responsibilities?
  • slim 5y ago
    How close are you to solving that ? And please would you share your progress :)
  • bambax 5y ago
    The brain hates to start things and loves to finish things. This can be hacked in that a working session should always leave something unfinished. Say you're writing a novel. Every writing session (but the first, obviously) should cover the end of the last scene and the beginning of a new one, AND THEN STOP, i.e. not finish the new scene. Your brain will want to come back to the work to finish it, which overcomes the friction of "starting" something new every time. It's easier said than done. It's surprisingly difficult to leave something unfinished at the end of each work session. But that's the trick.
    • xyzzy_plugh 5y ago
      The brain you describe is not the brain I possess. Starting is easy.
      • bambax 5y ago
        "starting" is ambiguous. I really meant "getting to work". Thinking about new things and maybe throwing down a few ideas is indeed easy and pleasant. But deciding to spend a few hours to move a project (instead of not) is what the brain hates. It hates commitment, and is very afraid of the opportunity cost.
        • jjk166 5y ago
          Nah, I'm with parent commenter on this. When I'm excited about something I have no problem diving into it for hours on end. But when I know that something is 90% done and it just needs to be tidied up, I will do anything other than working on it. Either everything from solving the hardest problem to being completely done happens in one sitting, or it never gets finished.
• snerual 5y ago
  Stateful, exaclty once, event processing without the operational capacity to run a proper Flink cluster. This thing needs to be dead simple, pragmatic and cheap/simple to operate and update. The only stateful part in our infra at the moment is a PG database. We are going to start work on this in a weeks, so I'm looking for some insights/shortcuts/existing projects that will make our lives easier. The goals is to process events from students during exams (max 2500studnets/exam = ~100k-150k events) and generate notifications for teachers. No fancy ML/AI, just logic. Latency of max 1 min. Our current plan is to let a worker pool lock onto exams (PG lock) and pull new event every few seconds for those exams where (time > last pull & time < now - 10s). All the notifications that are generated are committed together with a serialized state of the statemachine and the ID of the last processed event. Events would just be stored in PG. This solution is mean to be simple, be implemented in really short timeframe and be a case study for a more "proper & large scale" architecture later on. Any tips, tricks or past experiences are much appreciated. Also, if you think our current plan sucks, please let me know.
  • stingraycharles 5y ago
    (EDIT: just realised that you specifically mentioned stateful event processing, while what I describe below are two approaches for stateless, exactly-once event processing) Having had a few cracks at this problem, in my opinion using locks is the wrong approach. What you will want is: * split all input data in batches (eg batches of 10k records, or periodic heartbeats every X seconds, etc) * assign each batch a unique identifier * when writing data to the output store, store the batch id along the data; * when retransmitting a batch for whatever reason, reuse the same batch id and overwrite any data in the output store that matches this batch id. Obviously this becomes more tricky when you’re dealing with eg window functions or more complex aggregations. In this situation, I believe that an approach such as “asynchronous barrier snapshotting” works best. Every X seconds, you increment an epoch. While incrementing, you stop ingestion. Then you first tell the output source to create a checkpoint, then the input source to create a checkpoint, and once both have been checkpointed, you can continue streaming data again. Anyway, these are two approaches I’ve used over the years that work well. Explicit locks don’t work well in distributed processing, imho.
  • 5y ago
• needz 5y ago
  I’m working on OCR to recognize the scores on pinball machines. I have about a quarter million photos encompassing every model of pinball machine in the world but I just don’t have the know-how to accommodate all the font styles.
  • ad404b8a372f2b9 5y ago
    Have you tried an off-the-shelf solution like Tesseract? It works quite well if you do the recommended preprocessing.
    • needz 5y ago
      The preprocessing suggestions I see are to crop out everything except for the numbers and I don’t know how to do that programmatically. There’s many kinds of displays: rollers, 7-segment, dot matrix, and LCD. The preprocessing to increase DPI to 300 did not help when I tried Tesseract, unfortunately. It’s hard to achieve a good contrast between the numbers and the backdrop
      • ad404b8a372f2b9 5y ago
        There are a lot of other options and preprocessing methods you can use to get better results. It's hard to tell without seeing the picture but thresholding/binarization might help with the contrast. In order to isolate the text, the mode option also makes a lot of difference: https://tesseract-ocr.github.io/tessdoc/ImproveQuality.html#page-segmentation-method https://tesseract-ocr.github.io/tessdoc/ImproveQuality.html#... If that doesn't work you'll have to add a text localization model to your pipeline.
        • needz 5y ago
          Thank you for your guidance. I will investigate further
• iamwil 5y ago
  What are the resources to look at for designing the architecture of a scheduling system (for coroutines/threads)? Would I first look at how Operating systems implement it? Or how VMs like Erland implement it? Same question for effects systems, where would I look to understand how they're designed and the trade offs for their design decisions?
  • paulsutter 5y ago
    Everyone starts with callbacks, moves on to coroutines, and eventually ends up writing polling loops. Can take decades for each person to work through it. Erlang's messaging may have potential as another alternative
    • iamwil 5y ago
      By polling loop, do you mean like a while loop that checks a queue for work at a preset frame rate?
  • tommiegannert 5y ago
    Go is a modern take on coroutines (M:N threading), when everyone else seems to be going the async/await route. Though, I don't know how simple that scheduler implementation is (or if there are books about the internals). Since the Linux kernel has pluggable schedulers, that code might be well-structured for reading. Again, I don't know the specifics.
  • Matthias247 5y ago
    It indeed makes sense to look at on OS. You might want to start with a simple one, like FreeRTOS - which is more or less just a task scheduler at core. I would recommend not to start with coroutines in the beginning if your main focus is scheduling. In the end coroutines and async/await is about building a userspace scheduler on top of a scheduler that already exists in the OS, so you just get twice the amount of logic. However the schedulers used in userspace are often a lot more trivial than the OS ones, since they don't support preemption or priorities. Erlang might be the exception and an interesting thing to look into.
• betwixthewires 5y ago
  Some ZK way to prove that a piece of data was derived from some source, for example, proving a human fingerprint is unique identifying biometric data without showing the data itself or the person it is from and with no trusted authority.
  • cyberge99 5y ago
    Just thinking out loud here: I think it would require trusted sources (not necessarily a centralized authority) that have validate it and that you trust. You can prove two balls are different colors to a colorblind person by having them show you two balls of X color and proving to them that you can differentiate them (watered down example), but it requires validated externalities (ex, you can see colors they can’t). Defining the external validators is the hard part.
    • betwixthewires 5y ago
      If it did require a trusted source, do you think there would be a way for said source to 1) involve no human administration, 2) behave deterministically and 3) be resistant to attacks that could break either of the above 2? That would also solve the problem.
• deleted 5y ago
  [deleted]
• arrakeen 5y ago
  adapting the Sunday variation of boyer-moore algorithm to search backwards from point
• mhluongo 5y ago
  Community owned and operated FOSS alternative to MetaMask. Deciding which features to ship and what to skip is a bear — it's a big product design space
• xyzzy21 5y ago
  LOL Plenty of problems - none of them technical - all people problems!
  • jeanlucas 5y ago
    Yep, lots of problems are people problems.
  • ta988 5y ago
    I think these can be discussed as well. In the end thats the ones we suffer the most from. (edit: typo)
• jeanlucas 5y ago
  How can we make CTOs better? How does one CTO learn? How a community of CTOs can help create value to one another?
  • 29athrowaway 5y ago
    Why should CTOs only learn from CTOs? The average CTO is not as knowledgeable as you might think. For example, there are CTOs today building applications using no-code platforms, without any academic or technical background. Those people would have much to learn from an software engineering intern at any company. CTO is a job title, and each company can grant that title at their discretion. Or Technoking, or any other title you might think of.
    • jeanlucas 5y ago
      That's exactly the point, and mostly CTOs don't have much people internally to ask help from. I am in a community with some CTOs in Latin America and I see this struggle happen every time.
      • 29athrowaway 5y ago
        Latin America is a mess in many ways. VCs there suck, and do not understand that VCs are about inherently risky investments. They want the guarantees of a low risk business with the profitability of a high risk business. Leadership in Latin American companies also sucks. As soon as the company has any revenue, the leadership will go all out and spend it all on themselves on some extravagant lifestyle adjustments rather than reinvesting it on the company. And because of this, many companies stay small and mediocre without fulfilling their true potential. They waste their money on MBAs learning things they are unwilling to apply. Then, there is nepotism. Hiring from your family equates creating conflicts of interest, creating situations where relatives of key people do not need to comply with HR, do not need to be competent, cannot be fired and create an unprofessional atmosphere. Compensation in most Latin American companies sucks. They aspire to be like Silicon Valley startups and hang framed posters of Steve Jobs on their wall, but when it's moment to create compensation packages they grant zero stock and award zero bonuses, all while having the same work-life balance of a startup. They do not understand the key role of employee stock in company growth. Why the fuck would you work for a wannabe startup that doesn't give you any stock or bonuses? Or work for some entitled aggrandized clown that doesn't understand the simplest technical concepts? And the answer is: because business owners in Latin America have never had to care much about their workforce. Because of exploitation and their informal caste system, having a happy workforce has never been a requirement. That's why Latin America is undergoing a massive brain drain that will only get worse in the years to come.
• deleted 5y ago
  [deleted]
• red0point 5y ago
  I‘m trying to re-/sell cheap bulk object storage, by renting cheap dedicated servers (e.g. Hetzner), connect them using 10GbE and putting them into a big Ceph cluster. My problem is how to bill people for consuming object storage properly. Do you do it retrospectively and take the fraud risk? Are there any pre-existing platforms that do Ceph billing?
  • bob1029 5y ago
    Can you explain how the economics of what you are doing would be competitive with something like S3 or B2? I feel like there could be a market/margin here but there are a lot of numbers involved to figure out the specifics.
    • red0point 5y ago
      Just take a look at the egress pricing of B2 (10USD/TB) S3 (92 USD/TB) and then look at Hetzner's 1 EUR / TB. There is quite a margin there - same thing with the storage costs (23,5,1.5).
  • sosodev 5y ago
    This is what DigitalOcean did to create their block/object storage. If you're interested in doing this type of thing in a career capacity the storage team is hiring a lot of people right now. Feel free to reach out to me as I'm on that team. :) https://www.digitalocean.com/blog/why-we-chose-ceph-to-build-block-storage https://www.digitalocean.com/blog/why-we-chose-ceph-to-build...
  • oebboe 5y ago
    I'm currently thinking about starting a very similar project, would you like to talk about it and exchange some learnings?
    • red0point 5y ago
      Sure, how can I reach you?
• dzjin 5y ago
  I want to improve parts of online professional networking, specifically to be more about self-mentoring/shared learning, as opposed to sales connections. This is ever more important with the onset of remote hiring, remote work, and the isolation/depersonalization it brings to newcomers to the industry. There's also an "evil" momentum in remote hiring -- some companies _need_ asynchronous interviews to support their scaling and operations, and the general perception is that it's impersonal and dehumanizing. This made me think that if we preemptively answered interview questions publicly, then it'd empower the job seekers to have a better profile/fight back a dehumanizing step, while allowing non-job-seekers to share the lessons that were important to them. I've been getting decent feedback on my attempt at the solution HumblePage https://humble.page https://humble.page, the reality is that there's a mental hurdle to putting your honest thoughts out there.
  • sundarurfriend 5y ago
    This is a nice idea, to talk about and get thinking about these "soft" questions that people often struggle with. One feedback about the homepage: show a few examples of how people have answered questions, below the prompt. That's more helpful to get us thinking about our own answers, compared to a blank field. (Also, it's not clear what the percentages are meant to represent there. And I'm guessing the number next to the edit icon shows how many people have answered the question already? May need some UI tweaks on these.)
    • dzjin 5y ago
      Your guess is close! It is the total numbers, and the green/blue ratio indicate how many people answered the prompt publicly vs privately. My intention was to show the general comfort level of answering the prompt in public. Looking back, I wonder if I was being too quirky. I’m thinking the same on needing UI tweaks, I’m planning for major rearrangements. Thank you for your interest. Please feel free to reach out via the contacts if you’d like an invite.
• deleted 5y ago
  [deleted]
• pgroves 5y ago
  How to make png encoding much faster? I'm working with large medical images and after a bit of work we can do all the needed processing in under a second (numpy/scipy methods). But then the encoding to png is taking 9-15secs. As a result we have to pre-render all possible configurations and put them on S3 b/c we can't do the processing on demand in a web request. Is there a way to use multiple threads or GPU to encode pngs? I haven't been able to find anything. The images are 3500x3500px and compress from roughly 50mb to 15mb with maximum compression (so don't say to use lower compression).
  • primitivesuave 5y ago
    I would also be interested in knowing the answer to this. Currently we use OpenSeadragon to generate a map tiling of whole slide images (~4 GB per image), then stitch together and crop tiles of a particular zoom layer to produce PNGs of the desired resolution.
  • yboris 5y ago
    I'm unsure if this will help, but the new image format JPEG XL (.jxl) is coming soon to replace JPEG. It will have a lossless and a lossy abilities. It claims to be faster than JPEG. Another neat feature is that it's designed to be progressive, so you could host a single 10mb original file, and the client can download just the first 1mb (up to the quality they are comfortable with). Take a look: https://jpegxl.info/ https://jpegxl.info/
    • pgroves 5y ago
      This is a research university that moves very slow, so waiting two years for something better is actually a possibility (and prerendering to S3 works ok for now). I'll keep this bookmarked.
  • lightcatcher 5y ago
    I have no experience in PNG encoding, but found https://github.com/brion/mtpng https://github.com/brion/mtpng The author mentions "It takes about 1.25s to save a 7680×2160 desktop screenshot PNG on this machine; 0.75s on my faster laptop." which makes me think your slower performance on smaller images either comes using the max compression setting or using hardware with worse single threaded performance. Although these don't directly solve the PNG encoding performance problem, maybe some of these ideas could help? * if users will be using the app in an environment with plenty of bandwidth and you don't mind paying for server bandwidth, could you serve up PNGs with less compression? Max compression takes 15s and saves 35MB's. If the users have 50mbit internet, then it only takes 5.6s to transmit the extra 35MB, so you could come out 10s ahead by not compressing. (yes, I see your comment about "don't say to use lower compression", but no reason to be killed by compression CPU cost if the bandwidth is available). * initially show the user a lossy image (could be a downsized png) that can be quickly generated. You could then upgrade to a full quality once you finish encoding the PNG, or if server bandwidth/CPU usage is an issue then you could only upgrade if the user clicks a "high-quality" button or something. If server CPU usage is an issue, the low then high quality approach could let you turn down the compression setting and save some CPU at the cost of bandwidth and user latency.
• graphcolorer 5y ago
  I have a graph with weighted edges. I want to remove edges to make the graph colorable with N colors (e.g. N=40) such that the total weight of removed edges is minimized. If I'm able to solve this problem, that will complete a project I've been working on for years now to make a working keyboard for a person I know that has cerebral palsy.
  • sabageti 5y ago
    What are the needs in time?
    • graphcolorer 5y ago
      Can take as long as needed as it only needs to be colored one time. About 10,000 nodes, reasonably dense (about half the nodes will have 1,000+ edges).
      • sabageti 5y ago
        What did you try to color the graph ? CSP ? Classic backtracking ? With how many colors it must be colored ?
  • oneepic 5y ago
    Will the graph ever change? Is this a one-time computation?
    • graphcolorer 5y ago
      One time.
      • oneepic 5y ago
        Also... is it possible for you to add nodes? Ie split a N>40 node into two nodes.
        • 5y ago
• zakary 5y ago
  I’m working on a MEMS problem. I need to be able to 3D print micron (or even sub micron) scale features for a very high aspect ratio MEMS which uses electromagnets to actuate. Only problem is the leading candidate technology EFAB (electrochemical fabrication) only works with conductive materials. Does anyone know of a technology which provides similar abilities to EFAB but can also print features with non conductive materials?
  • fbeutel 5y ago
    Would something like Nanoscribe [1] work for you? The resulting structures are non-conductive. As part of our research we use it mostly for 3d-printing nanoscale optical elements, but we have also successfully used it for some mechanical support structures. [1] https://www.nanoscribe.com https://www.nanoscribe.com
  • Const-me 5y ago
    Not sure it gonna help, but some DLP 3D printers are very accurate, and most resins are non conductive. I know researchers were using one for micro-fluids applications. Not sure they can print sub-microns precisions, but 10μm is very real.
  • mikewarot 5y ago
    Could you use a conductive material, then deposit an insulating coating on top, or oxidize the material on top, making it an insulator? Aluminum can be anodized, then sealed. As can a number of other metals.
• fuzzfactor 5y ago
  Sheesh, there's so many things I'm miles from solving this was a tough one. It took a while but finally came up with something where it was actually closer than I thought. Thanks for the inspiration. Solved it.
• dzink 5y ago
  Working to enable users of https://www.DreamList.com https://www.DreamList.com to record audio of any length and see it transcribed, ideally at the same time as the recording, while the recording is also saved. The goal is for grandparents to save stories for loved ones and not worry about quality of the transcription - just talk. When the recording is saved, the transcription can be redone or tweaked if needed later, but the memory is not lost. DreamList is web and soon native apps, so WebRTC connected to a cloud transcription service is my first instinct, but there are benefits to native iOS apis as well - especially being able to keep share stories while listening to other streams also on iOS (families talking and digging into stories together). What architecture/transcription approaches would you suggest? Any gotchas you've seen dealing with similar problems (accuracy given accents, do we train our own transcription based on gathered data, etc)?
  • skoocda 5y ago
    I worked on this for a couple years during a previous startup attempt. I designed a custom STT model via Kaldi [0] and hosted it using a modified version of this server [1]. I deployed it to a 4GB EC2 instance with configurable docker layers (one for core utils, one for speech utils, one for the model) so we could spin up as many servers as we needed for each language. I would recommend the WebRTC or Gstreamer approach, but I wouldn't recommend trying to build your own model. It's really hard. Google's Cloud API [2] works well across lots of accents and the price is honestly about the same as running your own server. If you want to host your own STT (for privacy or whatever), I'd recommend using Coqui [3] (from the guys that ran Mozilla's OpenSpeech program). Note that this will likely be much, much worse on accents than Google's model. [0]: https://kaldi-asr.org/ https://kaldi-asr.org/ [1]: https://github.com/alumae/kaldi-gstreamer-server https://github.com/alumae/kaldi-gstreamer-server [2]: https://cloud.google.com/speech-to-text https://cloud.google.com/speech-to-text [3]: https://coqui.ai/code https://coqui.ai/code Edit: Forgot to mention, there's also a YC company called Deepgram that provides ASR/STT as a service, you could give them a shot: https://deepgram.com/ https://deepgram.com/
    • mkl 5y ago
      In my experience, Google's API completely fails when any slightly unusual vocabulary is involved (e.g. in this instance, grandparents talking about their past jobs), and tends to just silently skip over things. Amazon's wasn't much better with vocab., but at least didn't leave things out, so you could see problems. I don't have experience with any of these others, but I think for my purposes (subtitles for maths education videos) no one will have made an appropriate model yet.
• adriangrigore 5y ago
  Web site building complexity, for now I started with a simple static site generator https://mkws.sh/ https://mkws.sh/. Right now, I'm not using any package manager, no config files, only one language for templates (sh), and obviously HTML, CSS, Js. I eventually plan to develop a simple CMS based on the same ideas. Ideas and code are welcome!
  • chabad360 5y ago
    Have a look at stuff like Netlify CMS, or any of the already existing website builders like WIX or Weebly or squarespace Etc.
    • adriangrigore 5y ago
      Any headless CMS would work well my `mkws`, my idea is to build a smaller, simpler Wordpress that also comes with a tiny webserver, 0 config, no database, content stored as plain text files, just download and run. https://getkirby.com/ https://getkirby.com/ is closer in concept, but without the PHP dependency.
  • cxr 5y ago
    > (sh), and obviously HTML, CSS, Js Cut out the sh dependency and just use the "obvious" tech; make the site capable of generating itself, without reliance on any other tools.
    • adriangrigore 5y ago
      What do you suggest for templating?
      • cxr 5y ago
        ES6 backtick strings.
        • adriangrigore 5y ago
• kroltan 5y ago
  Economically sustainable and ethical monetization of user-generated-content games. The closest most known example of this kind of game nowadays is Roblox, but I'm thinking of things more like Mario Maker or the older-generation Atmosphir/GameGlobe-likes. Unlike "modding platforms" or simulators/sandboxes/platforms such as Flight Simulator, VRChat or ARMA, these games' content are created by regular players with no technical skill, which means the game needs to provide both the raw assets from which to build the content, as well as the tool to build that content. Previous titles tried the premium model (Mario Maker), user-created microtransactions (Roblox) and plain old freemium (Atmosphir and GameGlobe). I suspect Mario Maker only works because of the immense weight and presence of the franchise. Roblox's user-created microtransactions (in addition to first-party ones) seem to be working, but they generate strange incentives for creators, which I personally feel taints a lot of the games within it. (The user-generated content basically tends to become the equivalent of shovelware) GameGlobe failed miserably by applying the microtransaction model to creator assets, which means that to make cool content, creators have to pay as well as spend lots of their time actually building the thing, which means most levels actually published end up being the same default bunch of free assets and limited mechanics. Atmosphir is a bit closer to me so I find some more nuance in its demise, but long story short, essentially they restricted microtransactions only to player customization, however it didn't seem to be enough to cover the cost of developing the whole game/toolset. Eventually adding microtransactions to unlock some player mechanics, which meant that some levels were not functional without owning a specific item. --- In short, the only thing that can effectively monetize on is the game itself (premium model) or purely-cosmetic content for players. Therefore, to incentivize the cosmetics, the game needs to be primarily multiplayer, which implies lots more investment on the creator tooling UX, as well as the infrastructure itself. But this also restricts the possibilities for the base game somewhat.
  • jpindar 5y ago
    I'm surprised you didn't mention Core. https://www.coregames.com/ https://www.coregames.com/
    • abricot 5y ago
      It's a Roblox copy, and as such is already mentioned.
• ww520 5y ago
  I’m planning on implementing a transactional (ACID) key-value storage on modern hardware, i.e. SSD and large RAM. The technical choice on question is the type of index storage to use: b+tree, extending hashing, or linear hashing. There will be an appending WAL for batching the writes and for transaction support. A single checkpoint worker will apply the updates from WAL to the index storage periodically. The updates will be batched, to minimize the random seeks. Please help me evaluate the three indexing approaches, on the criteria of fast read, fast write, cache friendliness and ease of supporting atomic update during checkpoint.
• demarq 5y ago
  I'm trying to be the Amazon of real estate. If anyone is interested DM me
  • oebboe 5y ago
    That sounds interesting, sadly I couldn't find a way to message you on here, do you have any other way of contact?
  • Amir6 5y ago
    Please provide a way to contact you, ideally email!
• ark__n 5y ago
  Need help to decide the tools to be used for the below problem: The system is a bunch of batch jobs that are scheduled to run at different intervals. These jobs can be modelled as an acyclic directed graph of steps. They basically download files from vendors and map the rows inside them into a generic format (for generating reports). There are a lot of vendors and each vendor can have a different file format containing different fields -- hence requiring custom business logic to populate (map) the corresponding generic file (like aggregating fields, fetching values from DB, etc.). Also these vendors' files sometimes contain errors, or are dropped late for download, etc. -- failures can happen and these failed instances of jobs should be able to rerun. Existing system is built using Spring Batch and Spring Integration. The problems with the existing system are: 1. there are more than 200 jobs and most of them have their own custom logic during mapping -- cannot be generified 2. lot of manual work needed to onboard new vendors 3. jobs are synchronous and run only on one node, typically for lots of hours 4. rerunning jobs is a nightmare Dream state for this system: 1. Dynamically add jobs to the runtime using generic components that can be reused -- maybe through an API / UI 2. Preferably, multiple records from a single file be processed across distributed nodes to generate a single output generic file 3. Rerunning should be easier I am a noob to CS. I did a good bit of research for the past month. Found a few data-science tools in Python -- which is a no-no for a production system. Also, I know that the steps cannot be made generic after some extent since custom mapping logic is required for almost every vendor. But asking to see what is possible. Any help to point to prospective tools and technologies to solve the above will be much appreciated. Thanks
  • isoprophlex 5y ago
    Use Airflow maybe?
    • ark__n 5y ago
      Looks very promising. Can I add new jobs (tasks in Airflow's jargon) reusing my custom steps (operators in Airflow's jargon) during runtime? Also, is there something similar in Java, Go, etc.?
• marcosdumay 5y ago
  Yeah, this week I've restarted my scanning tunnel microscope that I'm failing to make work for years... The current one is a standard pair of long metal bars with the piezoelectric component on one end, with 2 screws, and a 3rd screw on the other end. My problem is that it doesn't matter how I design the thing, either the screws offer too little precision so I can't help but to crush the tip into the sample every time, or too little travel distance so I can't help but crush the tip into the sample when adjusting the coarser screws near the tip. This is the kind of thing that looks like a non-problem on the web, because everybody just ignores this step.
  • minhmeoke 5y ago
    Wow, this sounds very ambitious! Perhaps you could somehow attach the piezoelectric component or bars to a micrometer [1] which is designed for accurate and repeatable measurement? [1] https://en.wikipedia.org/wiki/Micrometer https://en.wikipedia.org/wiki/Micrometer
    • marcosdumay 5y ago
      Yes, I could. I believe the most straight forward design would be 3 micrometers directly supporting the piezoelectric component, with no levers. Yet, they are a bit expensive. I'm still not willing to budget all that, but I'm starting to consider it.
      • minhmeoke 5y ago
        Digital micrometers are expensive, but you can get analog ones on AliExpress [1] or other places for around $10. Of course, the precision may not be as good as name-brand (eg: Mitutoyo) tools. [1] https://www.aliexpress.com/wholesale?catId=0&initiative_id=SB_20210829113255&SearchText=micrometer https://www.aliexpress.com/wholesale?catId=0&initiative_id=S...
        • marcosdumay 5y ago
          Yes! Thanks a lot. Looks like the exact thing I need are micrometer heads, and some even come with nice threaded mounts.
• nickdothutton 5y ago
  I too am missing my 1990s forums experience. This feeling, and a particularly frustrating few minutes spent on LinkedIn prompted me to write something about it. I discuss some intellectual problems and solutions. https://blog.eutopian.io/building-a-better-linkedin/ https://blog.eutopian.io/building-a-better-linkedin/
• bastawhiz 5y ago
  I have a pile of mp3s and want to splice them together with a single ffmpeg operation. Essentially injecting multiple small audio files into a large one using time codes. I know there's got to be a way to do it, but I have yet to find a way to do it in a single operation instead of multiple passes.
  • dehrmann 5y ago
    Are they sequential, or are you splicing at arbitrary times?
    • bastawhiz 5y ago
      They might be spliced in at arbitrary times but multiple audio files may be spliced in sequentially at an arbitrary time. I'm not going to be upset if there's a millisecond of gap between them or whatever by just splicing at approximate time codes to make them sequential.
  • zamadatix 5y ago
    Some god awful combination in a complex filter using atrim to pull the pieces, adelay to set the positions in the output, and amix to put all of the output back together could probably do it. What that command may actually be is definitely open but that's probably the only way that would work as otherwise this is really 2 separate operations (chopping then merging) so there isn't going to be consideration for having it in a single premade flag. On another note if the goal is just to avoid files/writing to disk then a bunch of ffmpeg splices to named pipes as inputs to another ffmpeg command to merge them could do the same without the command soup.
    • bastawhiz 5y ago
      I'm very fine with command soup; the plan was to generate it with a script and run it directly. The input files are in a lossy format so I'd rather avoid having intermediate WAVs or double-encoding or something like that.
  • minhmeoke 5y ago
• udev 5y ago
  I am blocked on finding a good (defined below) way to determine whether a product description A and product description B refer to the same product. Imagine that a product description is a n-dimensional vector like: ( manufacturerName, modelName, width, height, length, color, ...) Now imagine you have a file with m such vectors (where m is in millions), and that not all fields in the vectors are reliable info (typos, missing info, plain wrong, etc). What is a good way to determine which product descriptions refer to the same product. Is this even a good approach? What is state of the art? Are there simpler ways? Here is what I mean by good: - robust to typos, missing info, wrong info - efficient since both m and n are large - updateable (e.g. if classification was done, and 10k new descriptins are added, how to efficiently update and avoid full recomputation)
  • alex989898 5y ago
    You could generate word embeddings for all natural language text fields and then do cosine similarity?
  • Fragoel2 5y ago
    Definitely some clustering method based on similarity of the vectors (there are many, pick a simple one to start)
  • ethn 5y ago
    Use a Minhash-LSH ensemble with pre-processing on the words to fix typos via Levenshtein. Tune parameters to get the best distance
  • gurgeous 5y ago
    I have worked on this problem many times, at many companies. I am working on it again, actually. Usually some combination of scoring and persisting results in CSVs for human review. (edit: I am at a desktop now and I can say a bit more) Here is the process in a nutshell: 1. Create a fast hashing algorithm to find rows that might be dups. It needs to be fast because you have lots of rows. This is where SimHash, MinHash, etc. come into play. I've had good luck using simhash(name) and persisting it. Unfortunately you need to measure the hamming distance between simhashes to calculate a similarity score. This can be slow depending on your approach. 2. Create a slower scoring algorithm that measures the similarity between two rows. Think about a weighted average of diffs, where you pick the weights based on your intuition about the fields. In your case you have handy discrete fields, so this won't be too hard. The hardest field is name. Start with something simple and improve it over time. Blank fields can be scored as 0.5, meaning "unknown". Hashing photos can help here too. 3. Use (1) to find things that might be dups, then score them with (2). Dump your potential dups to a CSV for human review. As another poster indicated, I've found human review to be essential. It's easy for a human to see that "Super Mario 2" and "Super Mario 3" are very different. 4. Parse your CSV to resolve the dups as you see fit. Have fun!
• jpeloquin 5y ago
  TLDR: What is the state of the art for one-shot or few-shot longitudinal (time-series) machine vision tracking of object boundaries with ~ pixel (~ 10 μm) precision? Specifics: I'm tracking the edges of the knee meniscus from time lapse video (~ 1000 frames) to measure its deformation under load. This is in the context of research to prevent osteoarthritis. Due to material rotation and irregular geometry, background edges that started off occluded come into and out of view over time. This tends to confuse both machine vision algorithms and human labelers. Because the tracking is for strain measurements, the tracked edge must be the same in all frames; therefore, the tracked edge is be a foreground vs. slightly less foreground edge of the same material (low contrast), not foreground vs. background (much easier). Only few-shot approaches are likely to save time because only ~ 20 specimens are needed to accomplish the immediate objective, and follow-up experiments will probably differ enough to require re-training. The current plan is to Google "few-shot image segmentation" and try things until something works or the manual labeling effort finishes first, but maybe one of you knows a shortcut. Work is also ongoing to bypass the problem by enhancing edge contrast or using 3D imaging, but machine vision would be the most cost-effective solution.
• ConfusedDog 5y ago
  I’m looking for methods (NLP or otherwise) that maps/generate a short description (few words or less) with a given format (e.g. 3 word tag with each word from a list of choices) from a paragraph or a sentence. Results would be stored and clustered base on their distance
• feurio 5y ago
  I've been working on a distrbuted Layer 2/4 Load Balancer (like Katran, but no C++ involved) that's mostly complete but now needs some testing with large workloads (1m+ concurrents, >50Gbps). Guinea pigs sought.
• zanek 5y ago
  I'm working on a different type of compression (for all file types). I am able to to get in the 10-20% range, but the speed to compress is to slow many times, or the compression doesnt complete at other times (I've been working on this for years). My personal website: http://danclark.org http://danclark.org I'm also working on a conversational search engine (using NLP) at http://supersmart.ai http://supersmart.ai
  • BudaDude 5y ago
    Have you looked into Middle Out compression?
    • zanek 5y ago
      Funny, I've actually had a lot of fun working on this compression software. It's a weird mix of needing it to be fast and hitting a compression threshold of it being useful. One of the best projects I've embarked on
• peripitea 5y ago
  Don't have anything to add right now but I like the idea of this thread and would support it becoming a regular thing.
• TeeMassive 5y ago
  A way to preserve and link factual data sets. Most reference to Wikipedia are dead links. Many legacy media will stealth edit articles or outright delete them. Original media files can be loss and after strange eons their authenticity will not be able to be asserted. It will soon be impossible to distinguish from deep fakes and actual original and genuine media. Some regimes such as Maoist China wanted to rewrite their past from scratch and erased all historical artifacts from their territory. There are strong pressure to create an Orwellian Doublespeak to erase certain words entirely from speech, books and records. With e-books now the norm it has now become legitimate question to ask if the books are the same they were when the author published them. Collaborative content websites have shown that they were not immune to subversive large and organized influence operations. I have set my mind to multiple solutions (even bought a catchy sounding *.org domain name!). Obviously it will have to be distributed as to build a consensus and thus it will have to rely on hashes. But hashes alone are meaningless so some from of information will have to come along with them, which in themselves are information to authenticate with other hashes. I was thinking that the authentication value would come from individual recognized signatories. Those would be a a mesh of statements of records. For example you might not trust your government, but you might trust you grandparents and you old neighbors who all agree that there was a statue on the corner of the street and they all link to each other and maybe link to hashes of pictures and 3D scans with links. Future generations can then confirm those links with other functional URIs. Something like blockchain technology seems an obvious choice but I have no experience with that (for now) but also there is the problem that it needs to be easily usable; therefore there is a need of a bit of a centralization (catchy domain name yay!) although any one could setup his/her own service for certain specialized subjects. Thoughts?
  • minhmeoke 5y ago
    One solution might be to collaborate with, build upon, and donate to an existing reputable organization like archive.org or Wikipedia which takes snapshots of websites. Given these snapshots, you could write manual extractors or build a machine learning system [1] to extract the main content of each page as plaintext. Then load these timestamped text file snapshots into git, which will give you a hash of the content and let you easily track changes. Push the git repo to a few places like github, bitbucket, and maybe IPFS where people can mirror it. [1] https://joyboseroy.medium.com/an-overview-of-web-page-content-extraction-5e0e2c62855d https://joyboseroy.medium.com/an-overview-of-web-page-conten... Alternatively you could use the built-in reader mode in Firefox or Chrome which do this automatically, but then you'd have to figure out how to maintain stability of the extraction algorithm between new browser releases.
• null4bl3 5y ago
  I am trying to load in raster layers from file in OpenLayers, but have failed so far.
• staz 5y ago
  this more like a tool suggestion request, because I haven't been able to find a solution in Google. I'm looking for a more advanced duplicate files finder in Linux, specially one that can handle folders. Most tool just return a list of duplicate files, but what I need is to also know is if whole folders are duplicate, or subset of others and have it presented in an elegant way t o resolve conflict. ofc everything is in a big folder and everything is a mess, bit of a hoarding collection of files that got accumulated over the years. I could probably code some stupid script myself, if I ever got the time (spoiler I probably won't) but I have no idea on how to present the result elegantly. So it would be nice if such tool already existed.
  • minhmeoke 5y ago
    https://meldmerge.org/ https://meldmerge.org/ might be what you are looking for. But if there are tools which can find duplicated files and directories which are named differently, I'd be interested in learning more about these as well.
    • staz 5y ago
      meld can compare two directory if they have the same structure and name for files but can't really tell if I inside my foot folder, I have a folder root/X which is also present as root/category/b/X. (except all the file in it have been prepended by "foo" somehow) For tools which can find differently names files you have for examples: fslint (gui) or fdupes (cli)
      • minhmeoke 5y ago
        That's true, meld is designed for comparing directories which already have similar structures. It seems like what we're looking for is content-addressable storage [1]. The theory behind it appears to be based on Merkle trees and cryptographic signatures [2]. IPFS already has an implementation [3] of this, and there are other implementations (borgbackup, restic, zpaq) listed in that link and in the content-addressable storage article. Disclaimer: I haven't used any of these yet, just found them a few minutes ago. [1] https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage [2] https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5 https://gist.github.com/mafintosh/464bb8f1451f22c9e5c5 [3] https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674 https://discuss.ipfs.io/t/ipfs-and-file-deduplication/4674
• verdverm 5y ago
  I'm working on generating code across the stack and languages from source of truth data models. Low code for devs. https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof Trying to reduce redundant tasks and simplify changes with minimal effort.
• deleted 5y ago
  [deleted]
• gpa 5y ago
  The problem - correct string matching at scale. I am aware of fuzzy string matching. The problem is that the two strings can be > 90% similar even if the difference is, for example, one digit in the year of manufacturing. My current solution is to represent the 2 strings as similar as I can based on the available information by transforming (wrangling) the data to match the data as close as possible and then applying constraints based on make, model and year (they should be the same). It works pretty well, but I am looking for a more interactive (human-in-the-loop) solution.
  • wizzwizz4 5y ago
    I'd just slap a GUI / audit logs on top. Show the intermediate data (the “wrangling”), show the computed similarities, show the conclusion (this met that threshold, and the other was equal, so it's category seven).
    • gpa 5y ago
      Can you elaborate on the technical details: which language, library or framework would you use?
      • wizzwizz4 5y ago
        Tkinter, probably. Or a web interface. Depends on what I'm doing, honestly – the answer will always be “whatever's currently being used”.
• deleted 5y ago
  [deleted]
• Const-me 5y ago
  How to create a good app.store for a smartphone OS? Users should be able to install whatever software they want. Similar for developers, they should be free to publish whatever software they made. Apple/Google approach is suboptimal because centralized point of failure. And they do censorship on their stores, both political and arbitrary. Linux approach is suboptimal because users don’t have keyboards to create these sources.list text files. Even if they had qwerty keyboards, I don’t like the UX, too hard to use. Traditional P2P like bit torrent + DHT is suboptimal because smartphones, would use too much electricity + bandwidth to be practical. So far, I’m thinking about developers-hosted binary packages, and existing code-signing infrastructure for authenticity and integrity (Verisign, Comodo, Digicert, these guys, up to developers to choose one). The configuration issue from Linux should be solvable with QR codes scanned by camera, plus a custom URI handler for the web browser on the phone. The main thing I don’t like about that approach — a store app on the device is a good UX from end users’ perspective. Yet it seems impossible to make one with that approach. I’m very far from being blocked on that yet, but I will face that problem eventually. P.S. I’m not going to solve security at that level. As Android store shows us, it’s borderline impossible even with Google’s resources. Modern mobile SOCs have enough juice to solve that properly, on the lower levels of the stack. Most of them support hardware-assisted virtualization. All of them are fast enough to run proper multi-user Linux, with security permissions and SELinux kernel module.
• palmtree3000 5y ago
  Json diffing. I haven't found any implementations I'd consider good. The problem as I see it is that there are tree based algorithms like https://webspace.science.uu.nl/~swier004/publications/2019-icfp-victor.pdf https://webspace.science.uu.nl/~swier004/publications/2019-i... and array based algorithms like, well, text diffing, but nothing good that does both. The tree based approach struggles because there's no obvious way to choose the "good" tree encoding of an array. I've currently settled on flattening into an array, containing things like String(...) or ArrayStart, and using an array based diffing algorithm on those, but it seems like one could do better.
  • te 5y ago
    At the risk of not being helpful: I have some json files that are updated weekly that I keep under source control in git. The week-to-week updates are often fairly simple, but git was showing some crazy diffs that I knew were way more complicated than the update. I soon realized that the data provider was not consistently sorting the json arrays; when I began sorting the json arrays by rowid everytime before writing it, the diff's were as straightforward as expected. I think I don't understand the problem you're encountering, because this solution seems too obvious.
  • loa_in_ 5y ago
    The problem is in the syntax of JSON itself. Use JSONL instead
• MAMAMassakali 5y ago
  I have a custom tlv encoding standard. It has 1 or 2 byte of header depending on type with the required info in the header. I have written an encoder and decoder for it, but now want to do it via ASN.1. But, I am not able to understand how to define header bytes and what rule each bit denotes in the header. Do I need to use ECN?
• desertraven 5y ago
  I find it hard to find communities for my ever-changing niche interests. I’m working on community discussion boards which exist at the intersection of interests. Eg. Mountain Biking/New Zealand, Propulsion/Submarine, Machine Learning/Aquaponics/Tomato, etc. The search terms for interests are supplied from Wikipedia articles which avoids duplicate interests and allows for any level of granularity. I find that key word functionality in search engines has degraded to the point that finding good content for niche interests is difficult. I’m hoping with this system I can view historical (and current) discussions around my many niche interest combos. I’ve got the foundation done, I just need some feedback/advice on whether I’m reinventing the wheel here, or if others share this problem?
  • thieu96 5y ago
    this kind of sounds like reddit to be honest, is reddit lacking something you wish would be there?
    • desertraven 5y ago
      Reddit is great for finding general topics - science, news, hobbies, etc. And the goal of this isn’t to replicate those more sweeping discussion boards because they’re great! My issue is that once things get more niche, the subreddits are tough to find - they could have obscure names, and even just learning about them often involves another user recommending. Plus creating a subreddit for every niche intersection isn’t ideal. With this I could zone in on exactly what I’m after, without sifting through all the unrelated stuff. With the Aquaponics/AI/Tomatoes, I’d be dealing with only that intersection point. Not the peripheral stuff.
• ammar_x 5y ago
  Please make this a monthly thread!
  • Amir6 5y ago
    Totally! Maybe if no one does we should do it every 1st of the month.
• myrryr 5y ago
  I've been dealing with needing to integrate an arbitrary area of a brane representing a density map. It's been slowing going getting all the maths done.
• maddynator 5y ago
  Trying to find line item based medical bills and laws that govern what information can patients obtain about medical procedure pre and post procedures. Goal is to bring transparency to medical bills and remove the unknown.
• sriram_malhar 5y ago
  I'm not sure I'm close to solving it, but I have an approach that I'd like some feedback on. I have a corpus of text in many Indian languages, which i'd like to index and search. The twist is that I'd like to support searches in English. The problem is that there are many phonetic transliterations of the same word (e.g the Hindi word for law can either be written as "qanoon" or "kanun"), and traditional spelling correction methods don't work because of excessive edit distance. My approach is this: Use some sequence to sequence ML technique (LSTM, GRU, ..., attention) to a query in English to the most probable translation and then use that to look it up using a standard document indexing toolkit like Lucene. (I can put together a training dataset of english transliterations of sentences to their original text) The problem is that I'd like the corpus, the index and the model to be all on a mobile. I have a suspicion that the above method won't straightforwardly fit on a mobile (for a few Gig of corpus text), and that the inference time may be long. Is this assumption wrong? How would you solve the problem? Would TinyML be a better approach for the inferencing part?
  • peripitea 5y ago
    I'm not sure I understand the problem specification. You want to be able to search "law", and find documents containing "qanoon" or "kanun", right? How does your proposed solution handle that? It seems like the approach with ML TL -> Lucene would still only find one of the two, unless your model is written to return a set of possible transliterations. Or are you saying your approach doesn't currently solve this part of the problem, and that's one of the things you'd like input on? Is the corpus the only data you have, i.e. do you need to use it for training and validation as well? In terms of the size of the data, if you want to store the corpus on the phone anyway, won't the index and model be relatively small in comparison?
    • sriram_malhar 5y ago
      No, sorry for the confusion. I want to be able to type “kanun” or “qanoon”, and have it infer the Hindi word “कानून”, which is an indexed word. It is not necessary that there is a one to one correspondence between words. Sometimes two english words may represent one hindi word, or vice versa. I believe I can build up a decent sized training/validation set, for example from Bollywood song lyric databases written in English, and mapping them to the Hindi equivalent (or Tamil, Bengali, etc). As for your last question, I dont know, since I haver implemented an ML model in practice. I saw a tutorial on Bert this morning, where a word has 768 features. That itself sounds huge, leave alone the model itself.
• ammar_x 5y ago
  I'm searching for a (reliable) open-source OCR tool for Arabic text. The best option I tried is Google Cloud Vision, but it's still not accurate enough, and it could get quite expensive for large tasks. Anybody knows a good software for that?
• berkeshire 5y ago
  Context: Not from the investment banking or trading background. Have been an investor / trader with modest gains. How does one solve for risk in the markets? As in, mathematically. How does one do short term predictions of prices, with a day or two as the prediction range, with a probability > 0.5.
• CRConrad 5y ago
  I've been thinking for a while about building a tool to generate ETL/ELT jobs for data warehousing. Yes, there are lots of such tools already, but I've become frustrated in one way or another with all that I've used so far (mainly with their bloated size and clunky "repositories" and inscrutable runtime engines). This "while" that I've been thinking is stretching out -- the other day I stumbled across my earliest vague musings on the subjects, and noted that they are a couple of years old already, so I'm beginning to think it's time to stop thinking and start building... For various reasons -- mainly familiarity, FOSS ecosystem, and cross-platform compatibility -- I'm going to try to implement this in Free Pascal / Lazarus. There is one kind of component I'm definitely going to need, and if there were a ready-made one I could use in stead of building one from scratch, it would save me a lot of time and effort. I've looked around online, but so far haven't found "the perfect one". So, my question is: Can anyone recommend a good FOSS graphical SQL query builder component -- i.e, one which presents tables and their columns to the end user, so they can specify joins and filters by clicking and dragging, etc -- for Lazarus (or in a pinch Delphi, to port to FP/L)?
• ElectricMind 5y ago
  I have to wake up and pee 2-3 times a night. I put pillow under bed on my leg side and make leg side up. It is working so far. Every 4-5 hours from every 3 hours. I also trying Kegel exercise too. I can't stop drinking water before bed because I have acidity too and I can not sleep with dry throat :)
• roland35 5y ago
  I am trying to design a solution to something I encounted a lot during my hardware development days - keeping track of inventory of small and large parts alike. Basically a shelving system with an app which a user can search and the shelf would illuminate the correct bin. My problem is, would this be a viable product for companies? It was always a hassle when I worked in labs but I am not sure if it is a big enough problem to design an entire IoT product around.
• lsferreira42 5y ago
  I have two app/websites ideas, one is a way to track your food and alert you when your food can spoil ( and sugest recipes with what you have in your house ) And a learn helper website/app that will give you paths to learn thins and register, track and help you through this process The first one is on the planning side, the second one already have some pages written in python/django
• kjrose 5y ago
  I've been working on applications of network flow theory and ecological measures to macroeconomics to determine mesoeconomic scale measures for use by decision makers and businesses. These measures use certain properties to determine both stability due to sudden shocks to the economic system as well as what various policy actions will have on the overall health. This includes entirely different ways of visualizing free trade, etc. I have made some significant progress in the past year, but I am running into this headache where the papers I need are not easily accessible (or cheap) since I'm not an academic and I'm literally doing this as a for fun side project. I've been trying to find a university or college willing to take me on in some capacity simply to let me get access to academic catalogues, but I am not having that much luck unfortunately. The best results recently though include some really fascinating stuff that comes from statistical measures that give clarity to how a region would respond to being isolated for any period of time.
• frozencell 5y ago
  How memes could be another Bitcoin.
• ID1452319 5y ago
  There is a juxtaposition in the UK job market. We have millions of people working in low-paid precarious jobs in retail, food service, warehousing etc. while simultaneously companies complain that they cannot recruit into highly-paid, skilled roles due to a lack of candidates. Given that you can study Introduction to Computer Science from Harvard University, online, for free and in your own time, it seems like the barriers to building skills is lower than ever. However, many people are put off or intimidated by the idea of studying such a course. My solution to this is some kind of mentoring, either 1-to-1 or more likely in small groups. However, this is very resource intensive for my idea to scale. I'd be very interested to hear how others might approach this, both the mentoring or the underlying encouragement to study.
• rkohn33 5y ago
  I want to leverage data provided by the healthcare system and create a machine learning algorithm that monitors the patient at all times by leveraging Iot data, vitals, location, all possible identifiable data without breaching hipaa etc...