Ask HN: Why is ChatGPT so bad at electronics?
I feel LLMs like GPT-4 could be a perfect tool for tedious, time consuming tasks like searching through datasheets for components with specific characteristics, but in its current form that's not the case: ChatGPT for instance can list components of quite obscure families and seem to have a quite a accurate general idea of their purposes, but when it comes to important details it just fails spectacularly every time.
Is there any company trying to improve this application of language models?
ChatGPT can be hilariously bad at less common things, for example, I asked it for the uses of polyurethane foam in my native language, and it suggested it would be great for decorating cakes.
This applies to many fields. It will come up with plausible looking but wrong answers and keeps apologising if you correct or point out the mistakes.
I've seen it with statistics as well, asking it to implement some things in code. You'll get working but mathematically wrong code.
I think ChatGPT is just bad at whatever domain you have expertise in.
Talk to it about something you don't know about, and you'll think it's really good technology ;)
"Everything you read in the newspapers is absolutely true except for the rare story of which you happen to have firsthand knowledge." - Erwin Knoll, allegedly[1]
[1] It's widely attested to him, including on Wikipedia and in an aside in an old NYTimes article [https://www.nytimes.com/1982/02/27/us/required-reading-smith...], but I couldn't track down the original source to verify it. It's possible it's falsely attributed, but on the other hand he wasn't super prominent -- editor of a major magazine for progressivism, called The Progressive, yes, but no Mark Twain -- and so there'd be little incentive to say it was his when you could call it Twain's.
So it shows an electronic form of Gell-Mann Amnesia?:
https://www.epsilontheory.com/gell-mann-amnesia/
Ah thanks for the link. I’ve been trying to recall the name of the phenomena for ages and it’s always been just on the tip of my tongue.
As an electronics engineer, I have tried it for such tasks, without success. I specified requirements (only the key/rarer ones, typically 1 or 2) and asked it to find components. It failed miserably, typically just insisting that some related but much more common component satisfied the requirement. More and more apologetic as I tired to guide/coax it along. I know that there are a few components available that satisfy the requirement, as well as several hundred that do not. And I know that the information is in digitally readable PDF files (as opposed to scans).
This specific failure might be s kind of averaging problem, where common answers around the general theme are preferred over more specific (and correct). LLMs can also fail completely at trivial concepts such as negation, or separating between "Y above X" and "Y below X".
The thing about data sheets is you have to watch out for your own assumptions when reading them. If it doesn't explicitly say it'll do X... it won't, no matter how common it is in other parts of the same type.
It might help but you have to be the backstop when it comes to the final call. Measuring the false positive/false negative rate could be tedious, but it's important to have a good estimate of, in order to use it wisely.
Yeah, I see what you mean, like if I want to find for instance "a sub milliampere X" but in the pdf it only says uA then it would be impossible for an LLM to suggest that
How much would you pay for this feature?
Is typing your requirements that much easier than going through traditional search filters at Digikey?
I asked it all sorts of specifics about how to use my esp32 and it does surprisingly well
This is unsurprising since from what little I know about electronics, I know the ESP32 is pretty common. I know very little about electronics. So if I know it's common, it's for sure in the training data of ChatGPT.
Do in context learning. Gather a huge sheet of specs for components you want to use, put it at the top of your chat, and then ask questions.
Have you tried ChatGPT apps specialized for electronics, such as https://chatgpt.com/g/g-6PTe1fb3X-electronics-and-circuit-an... ?
Here's the "app":
You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. You are tasked with answering questions and providing assistance in the domain of Electronics and Circuit Analysis. You will apply structured and concise explanations while incorporating relevant academic and technical references from uploaded materials. Your goal is to ensure clarity, accuracy, and technical correctness in your responses, especially when dealing with advanced concepts. Always follow the structured format requested by the user.
No, just straight chatgpt and claude but I will take a look, thanks