Part 1 of this article looked at the challenges and unavoidable uncertainties facing so much of medical research, and why all-important replication of experiments is so difficult to accomplish. This part looks at the corresponding situation for engineering research and product assessment.
After thinking about the medical-research situation, I compared the issues to those related to the many tests which scientists and engineers perform on their research, innovations, and products at all stages ranging from the R&D phase through prototype, production, and even formal release. Let’s be honest: it’s far, far easier, and more reliable to run an array of tests to evaluate a new device or product than to do the equivalent in medical research. Think about it:
- you know quite a lot about the unit under test: how it was made, the performance of its constituent elements, and how it was assembled; its “provenance” is reasonably clear. That’s quite different from having a fuzzy medical history. Yes, there are cases where a subtle change in how a component was made affects the unit-to-unit results, but that’s not the general case.
- you don’t have to worry about the subject not following the planned research protocol, skipping medications, having pre-existing conditions, eating/not eating as directed by the protocol, not fully reporting on activities for whatever reason), and so on;
- you can perform fairly detailed tests on a single unit and get meaningful results; you can also build up a pilot run of identical units to collect data which help take tolerances and variances into account;
- you can model the unit before and after tests to varying degrees of fidelity depending on how much time and effort you want to invest. Even if the model is not perfect – and they never are – it still is usually fairly good and can be made better as more data is collected;
- even when the tests are sophisticated and there is a legitimate possibility of errors due to the concept, set-up, or execution, there usually are ways to cross-check the results and get some level of confidence in the credibility of the results;
- while it is the usual case for other engineering researchers to try to replicate your efforts in many cases, it can be done and is done in special ones. For example, soon after Theodore H. Maiman at Hughes Research Laboratories demonstrated the first ruby laser at a press conference (coinciding with his very basic one-page paper in the “letters” section of Nature “Stimulated Optical Radiation in Ruby”), researchers at other institutions repeated his work and confirmed it was not a mistake or misleading “light show.” Furthermore, even when others do not confirm the work directly, they build on it and extend it as they advance the technology. By doing so, those subsequent researchers do confirm the original work.
- Finally, most changes in science and engineering tests are generally reversible, which provides another tool for confirming a hypothesis and results. In contrast, it’s very hard to do that with medical research, as you often can’t undo the effects of specific actions (“so it wasn’t the kidney, after all, we’ll just put it back”). Even if you stop giving a certain medicine to attempt to return to the baseline, its effects may often linger or have done irrevocable damage. An option may be double-blind tests, but they are hard to set up, time-consuming, costly, and still have issues between test and control groups.
Stepping back and thinking about the bigger picture, you have to admit to the reality: implementing engineering and science tests, gathering reliable data, evaluating that collected data, and extrapolating it to meaningful conclusions with a high level of confidence is much, much easier than doing many types of medical research.
Many years ago, I was in an informal, spirited debate with another engineer. He maintained it was as hard to be a good engineer as it was to be a good doctor and that the requirements for both were equally difficult. I only partially agreed. Sure, there’s a lot to know for both, but the engineering profession generally doesn’t have an equivalent to the structured residency cycle, and instigating changes is easier and less risky. While engineering mistakes at the prototype bench may kill a circuit board (a proxy for the patient), that’s very different than an incorrect medical diagnosis and subsequent action.
I maintained at the time that many of the ambiguities and risks in medical research are absent in much of the engineering equivalent and would actually be at odds with the kind of certainty on which engineers thrive. The Covid-19 virus, its data, and the related challenges have made me even more certain of that view.
