14 ms·
AB07-USB3FMC: The $1.65k eval board they can't be bothered to test
- aidenn0 6y agoEval boards are often more buggy than production boards for the exact same reason they are more expensive: volumes are low, so engineering work must be amortized over a much smaller number of units.
- chmod775 6y agoThis explains the cost, but why are they supposed to be more buggy? Should you have time to test those individually if there's just a small number of them?
- pdpi 6y agoIf scale is small enough you have to tackle the problem from both ends: cut costs and raise prices.
- bryanlarsen 6y agoIn low volume you don't set up automated tests, nor do you test the tests.
- dvdbloc 6y agoNor do you ever actually even run all the tests... at least that's how it has appeared to me from my experience!
- zrobotics 6y agoIn college I worked as an inspector at a plant that made aerospace boards. Each board was tested multiple times, there were inspectors after almost every step, to the point that each board was inspected at least 4 times during assembly, and then 2 separate full tests were done on every board. No sampling, each and every board was tested. Speaking to the engineers, our projected defect rate was <1%, and we achieved that. However, for rough numbers, this 1.6K board would likely cost >10K to achieve that level of reliability.
- variaga 6y agoIt's not just the small volume. Having worked on a number of ASIC + eval-board combos, there are several issues: 1) Eval board design typically begins before the ASIC design is finished. This means eval boards are frequently subject to last-minute design changes (because something in the ASIC changed) and/or intense schedule pressure for the design to be completed and the boards manufactured between the ASIC design freeze and when the ASIC comes back from the fab. Last minute design changes and high schedule pressure are not conducive to high-quality engineering. 2) Eval boards frequently have multiple purposes. The primary focus of an "eval" board is likely to be test platform for the ASIC, and customer evaluation is a secondary goal at best. As an external customer of the eval board, you're not even the _primary_ customer - the primary customer is the internal testing team. If "easy for the test team to use" and "easy for an outside customer to use" ever come into conflict, the test team is going to win. 2b) Because eval board's real target customer is a team within the company that has access to the ASIC documentation, preparing external documentation is a low priority. And since eval boards can indirectly expose embarrasing bugs in the ASIC, design secrets, etc., the external documentation is either heavily redacted, gated by onerous NDAs, or both. 3) Eval boards aren't the product - the product is the ASIC - and selling eval boards is rarely profitable on net, so there's pressure to keep the costs down. In the same vein, the PCB team for a semiconductor company is considered lower status within the ASIC company (if they're even part of the company and not contractors or an outside design house, which is also very common). This doesn't mean they're less skilled, necessarily, but it does mean they have fewer resources and less influence within the company, so "eval board" issues are going to be low priority. 3b) Because eval boards are a net loss, the incentive to go back and clean up any issues caused (1) and (2) is low to non-existent. 4) Because the eval boards _never_ go to high volume production, the eval board design team frequently has no experience or motivation to do proper DFM (design for manufacturing) or DFT (design for test) on their PCBs. Assembly is frequently manual because the volumes are too low to justify setting up (and debugging) an automated assembly line, leading to high product variability.
- marshray 6y agoAlso: 5) The eval board price is intentionally high to filter out non-serious users who're not engineering with a plan for eventual high volume production. Sales reps may just eat the cost of some dev resources if it enables a volume sale, or get it reimbursed as promotional expenses. I met with a field engineer for a major chipmaker. I asked him about the samples that he tossed to our group, he said he usually just got them from DigiKey.
- dylan604 6y agoit would be interesting to see if the same "wrong" resistor was used on multiple boards in the same position, or if the stack of resistors loaded into the robot just had one out of place resistor. if it was just a one off, thinking of the odds that one component out of the many on the board, the one board found it's way to the one guy that is skilled enough to not only notice, but do find out what, and have the proper component to swap out. I'd be buying lottery tickets if it were me.
- jjoonathan 6y agoI'm not convinced it was the wrong resistor, it could have had the correct value until someone (not necessarily OP) grabbed the board wrong, dropped it, or dropped something on it, and scraped some of the conductive layer. In any case, having covered circuit boards is a critical part of the reliability equation in consumer electronics. In an EE environment, component-level access and visibility can make it worthwhile to deal with naked boards, but you inevitably drop a few nines of reliability.
- fest 6y agoThe scenario you describe is very unlikely (the conductive part of chip resistor is enclosed in protective coating). What is a lot more likely is one of the following: * Engineer sent incorrect assembly drawing (sometimes when components are shuffled around with silkscreen/assy drawing layer hidden, the label for component can end up in unexpected place). * Someone manually updated the part numbers from one revision to another and missed the one resistor. * A new engineer joined a project, and after minor unrelated change to project, regenerated the fabrication outputs as one should. But he was not aware of changes somebody else did in BOM file by hand. * PCB assembly was performed by hand for prototype and contained an incorrectly placed part, but the assembled PCB ended up being shipped because of complete lack of production planning, somebody in sales selling stuff before it's ready and this being the only option to avoid contract cancellation. I have personally witnessed every one of these and then some. Yes, product engineering sometimes really is that shoddy and people being people stupid things.
- 6y ago
- ezoe 6y agoI think I read it somewhere that somebody ended up using consumer ARM laptop for a build server because all the ARM board in the market are crap and can't sustain 24 hours heavy load while consumer grade laptop at least get tested sufficiently to keep working under the heavy load.
- centimeter 6y agoAbsolutely. As an embedded engineer, 90% of ARM boards (including and especially the popular Raspberry Pi) are complete dogshit. Barely (or not) functional, absurd design decisions, etc. A hilarious example - to deal with the poor reliability of the RPi, you might think to use its watchdog timer to reset the device if it locks up. But guess what? The WDT in the crappy Broadcom chip on the RPi doesn't work! It's supposed to be there, and it's exposed to the kernel, but it doesn't actually do anything.
- my123 6y agoThere are reliable choices like boards using SoMs (from NXP or NVIDIA), or even the Raspberry Pi Compute Module. For example the Honeycomb LX2K at https://www.solid-run.com/nxp-lx2160a-family/honeycomb-workstation/ https://www.solid-run.com/nxp-lx2160a-family/honeycomb-works... or the NVIDIA Jetsons at https://developer.nvidia.com/embedded-computing https://developer.nvidia.com/embedded-computing. Those options have the advantage of using SoMs so that you can customise the boards for a higher-volume run too. The regular RPi on that point isn't exactly ideal, but depending on the use might be quite _okay_.
- ahartmetz 6y agoHere in Germany, Toradex ARM SoMs are quite popular for industrial applications. The hardware is reliable and the software for it (Windows whatever CE is called now and Linux BSPs for Yocto) is pretty good on release and well maintained for many years afterwards. The modules are not exactly cheap for the performance but also not super expensive, generally in the 50-150€ range.
- 6y ago
- SV_BubbleTime 6y agoI mean, inconsequential but $1.65k is more characters than $1650. Is there some always us “k” writing rule I don’t know about?
- svnpenn 6y agoYeah, I would use whatever is shorter
- segfaultbuserr 6y agoIn engineering notation, it's usually preferred to use the appropriate SI prefixes (pico, nano, micro, 1, kilo, mega, giga...) in the unit of measurements, each SI prefix represents three orders of magnitude (1000). When the number is greater than 1000, you always upgrade your unit, so that the numerical value is never greater than 1000 [0]. Instead of "1500 Hz", You say "1.5 kHz". OP is an engineer and I guess she has been trained to always do it. [0] These rules are not always enforced, and there are alternative notations, but the 1-1000 rule is the most common convention, writing "3500 MHz" instead of "3.5 GHz" is just awkward.
- myelin 6y agoNot the case in Kate's usage, but there's also the precision thing: 1.65k means 1650 +/- 5, whereas 1650 or 1.650k means 1650 +/- 0.5.
- nickff 6y agoMost engineers get used to using 3 significant figures, because the rest is usually noise; some will 4 'sig-figs' if the leading number is a 1.
- userbinator 6y ago...unless you work in RF, where 3500 MHz and the like are used for precision.
- busrf 6y agoI’d bet she was probably thinking about resistor values while writing the title! You would always say 1.6k resistor and not 1600kohm resistor
- 11thEarlOfMar 6y agoI once had a manager tell me, with a perfectly straight face, "Our customers are the testing department."
- mattlutze 6y agoI am curious why the author abbreviated the price of the board in such a way that both didn't remove any characters from the sentence and made the number more obscure.
- segfaultbuserr 6y agoIt's written in engineering notation. https://news.ycombinator.com/item?id=25163303 https://news.ycombinator.com/item?id=25163303
- mattlutze 6y agoThe what isn't the question dear friend, the why is. And I'm confused why they'd make an unnecessary choice to decrease readability.
- segfaultbuserr 6y ago> The what isn't the question dear friend, the why is. If you followed the link, you should've seen that I wrote "OP is an engineer and I guess she has been trained to always do it", so I thought it would be all you need to understand "why", isn't it? But if not, to elaborate - The likely explanation here is: Engineers are trained to do it in their jobs, and for someone from an engineering background, it's possible to "make this choice" automatically without even realizing it, and for them, it's totally a natural number without any readability issue. In the same sense, if you see a software developer who expresses a number in the power of 2 even when it's unnecessary, it may not be the most appropriate decision, but this type of behavior is totally understandable. You don't have to be a rocket scientist to see why.
- okl 6y agoSo what was the faulty resistors value?
- Teknoman117 6y agoTail of the article shows 396 ohms rather than 50 ohms.
- DanBC 6y agoShe says she measured it at 394 ohms. The closest standard value is 390ohms. I wonder where else that's used on the board, and if this is a pick and place machine error.
- okl 6y agoYep, could be. Could also be some BOM error in their part library if that problem is present on every board of a certain revision.
- ChuckMcM 6y agoI saw her take on this on Twitter and flinched. We had a similar experience with a > $5K eval board from Xilinx that we were basing a product on, and a customer wanted to start software development before the product was ready so we sold them a setup that was equivalent to our product in design (and embedded the Xilinx board) They found a problem which turned out to be a design defect on the eval board so we went to Xilinx for a warranty replacement, they refused saying they expressly disclaim any "fitness for purpose" on their eval boards and their terms of service disallow using their eval boards in a product. I rather rudely asked them why they charged money for them then. A very annoying experience.
- ChuckNorris89 6y agoYep, the hardware industry is notoriously bad for making and supporting quality eval boards. Heck, where I worked, most eval boards ware usually side projects given to summer interns or new grads or outsourced to India.
- ChuckMcM 6y agoOuch, this just makes me sad.
- R0b0t1 6y agoIn Kansas there is a law explicitly dealing with this: You can not disclaim an implied warranty or suitability for a particular purpose. In the broader context of the entire US, this is just a restatement of common sense law that is already on the books. I bought the product to fulfill a need. If it can't do that, then you don't get to keep the money. Nobody hands money over for some vague, nebulous promise of something in return.
- ChuckMcM 6y agoThat is pretty neat, I thought California's warranty laws were pretty strong but the lawyer suggested we wouldn't win if we sued. (not that we'd likely sue the only supplier of a part we need but you know, "in principle"). Does the Kansas law apply to software as well? That would be a neat trick. There was a national effort to get this level of support in the UCC (universal commercial code) but as I recall it was unsuccessful.
- gumby 6y agoThis kind of thing happens all the time. Intel sent us an eval board that wouldn’t boot. After various people screwed around with it I finally looked at the schematics and saw a pullup capacitor on the UART Vcc. A glance at the board and indeed there was a cap between Vcc and the chip supply. An easy workaround but how the hell did this pass? So the board was booting up fine, you just couldn’t talk to it. We excited the chip wasn’t going to survive and indeed, it didn’t.
- unwind 6y agoUh not to sound like a ton of bricks, but what is a "pullup capacitor"? You mean the connection to the chip's Vcc was a capacitor (effectively blocking any DC current, and not powering the chip), instead of just a trace?
- gumby 6y agoYes, clearly someone had meant to put a resistor there but instead spec'd a cap, making an open circuit instead. Shorting out the cap solved the problem. Since they'd shipped these boards for a while it was pretty likely nobody was using them. Since nobody wants to design around a part likely to be cancelled due to poor performance in the market...we figured nobody would be relying on this part. I called it a "pullup capacitor" because it was where you'd expect to see a pullup resistor and thus the name is funny.
- jhallenworld 6y agoFWIW: I had success using Cypress FX3 for USB 3.0 (connecting it to a Zynq FPGA using parallel interface). It's a weird chip though- you have to design the parallel protocol using a state machine GUI tool (I did not like their default design). It's an ARM CPU, so you also have to be prepared to program it. There is a simpler to use FTDI chip for USB 3.0 also, but have not tried it.
- voltagex_ 6y agoWhat are Cypress' tools like? I've only played with the FX2 and it's reverse engineered to the point you don't need the vendor tooling any more.
- jhallenworld 6y agoSo there is the ARM firmware and the USB host software. For Linux they provide a Cypress wrapper for libusb and some examples, that was straightforward. For the ARM firmware, there is a short document called "Using the FX3 SDK on Linux Platforms"- it was pretty straightforward. It's arm gcc, plus a qt-based program called "cyusb_linux" to interact with the FX3's bootloader (you program it over USB). https://www.cypress.com/documentation/software-and-drivers/ez-usb-fx3-software-development-kit https://www.cypress.com/documentation/software-and-drivers/e... You can use Eclipse if you want (they have some kind of support for it), but I just used the Makefile in the example projects. The ARM firmware uses ThreadX RTOS, but the actual application is pretty small (and the space is very limited as I recall). The only stupid thing is that they have this thing called GPIF-II designer, which I think is a Windows-only tool. The final output is a table in a C header file. I think it's a .NET program, so I think it would be easy for them to support it in Linux, but they don't.
- voltagex_ 6y agoNow, if only I had an excuse to use one. Thanks!
- kenz0r 6y agoI had a good one with a thesis student I supervised. The security micro she chose was designed to sit between SPI Flash and the main SoC, so a big feature was hardware accelerated SPI slave support. Guess what feature didn't work on the dev board...