Embedded Systems in Harsh Environments: What Actually Fails

A board that works on the bench and works in the climate chamber can still be dead in a roadside cabinet in eighteen months. Not because anyone specified the wrong part, but because the failure modes that matter in the field are mostly absent from the datasheet.

Harsh-environment embedded design is often discussed as though it were a component selection problem: choose industrial-temperature parts, add conformal coating, specify a higher IP rating, done. That is the easy half. The expensive half is everything that only shows up after a couple of thermal cycles, a power event, or a winter.

Here is what actually fails, in roughly the order it will cost you money.

Temperature range is the least interesting thermal problem

Industrial parts rated to −40 °C to +85 °C solve the question of whether the silicon survives. They say nothing about the two thermal effects that actually break deployed equipment.

Thermal cycling. A cabinet on a motorway gantry may swing thirty degrees a day, every day, for a decade. Each cycle works the solder joints, the connector contacts and the board itself, because everything on it has a slightly different coefficient of thermal expansion. Failures appear as intermittents first — a link that renegotiates, a sensor that drops out on cold mornings — and the intermittent stage can last months before anything reads as broken.

This is why a device can pass every environmental test and still fail in year two. The tests prove it survives the range. They rarely prove it survives ten thousand traversals of that range.

Self-heating in a sealed box. Passive cooling assumes air movement that a sealed IP66 enclosure does not provide. The internal temperature is not ambient; it is ambient plus whatever the electronics dissipate, minus what the enclosure walls can shed. In direct summer sun on a south-facing cabinet, the delta is routinely larger than designers expect, and it lands on components already close to their derating limit.

Condensation is the failure that better sealing makes worse

This is the counter-intuitive one, and it catches experienced teams.

An enclosure is sealed on a humid afternoon. Overnight the temperature falls, the trapped air cools below its dew point, and moisture condenses on the coldest surfaces inside — which are usually the metal parts and the board. In the morning it warms and evaporates. Repeat daily.

A better seal does not prevent this. It prevents the moisture leaving. The result is a slow accumulation that produces corroded connector pins, dendritic growth between fine-pitch pads, and eventually leakage paths that look exactly like a firmware bug: the device works, then does not, then works again after a warm day.

The mitigations are unglamorous and belong in the design, not the field:

• Breathable membrane vents that equalise pressure while blocking liquid ingress, so the enclosure can exhale rather than trapping wet air.
• Conformal coating, applied with genuine attention to what is masked — connectors, test points and anything that will be serviced.
• Positioning heat-generating parts to keep the board above dew point, rather than optimising purely for lowest temperature.
• Choosing connectors for their plating and sealing, not their price. Connectors fail far more often than silicon.

Power is where equipment actually dies

Field power is not bench power. It sags when a motor starts, it carries switching noise from equipment sharing the supply, and it is interrupted more often than anyone plans for.

Brownouts are more destructive than blackouts. A clean loss of power is a safe state. A supply that sags to a marginal voltage and hovers there can leave a processor running below its specified minimum while peripherals behave unpredictably — which is precisely how flash gets corrupted mid-write. A device that will not boot after a power event has usually been damaged by the shape of the event, not the loss.

Design for it explicitly: a supervisor holding reset until the rail is genuinely stable, a filesystem that assumes power will be removed mid-write, and configuration stored so that a corrupted write cannot leave the device unbootable. Assume the device will lose power at the worst possible moment, because across a fleet and a decade, it will.

EMC failures look like software bugs

Motors, contactors, variable-speed drives, radios and welding equipment all inject energy into cables that were not designed to receive it. What follows presents as a software problem: a bus that locks up when the pump runs, a sensor reading that jumps when a gate opens, a comms link that fails only at one site.

Teams lose weeks to this because the symptom is in software and the cause is in the physical layer. The tell is correlation with an external event rather than with an input value or a code path. If a fault tracks something happening in the room rather than something happening in the data, stop reading the code and look at the wiring.

Cable routing, shield termination and galvanic isolation on anything leaving the enclosure are design decisions. They are close to impossible to retrofit once an estate is deployed.

The failure that costs most: you cannot tell what happened

Every problem above is diagnosable if you have evidence. In the field you usually do not, because the device that failed is the device that was recording.

For a deployed estate this is the difference between a diagnosis and a guess:

• Log to storage that survives power loss, and treat the log as a designed artefact rather than debug output left switched on.
• Record the environment, not only the application — supply voltage, internal temperature, reset cause, uptime. Reset cause alone separates a watchdog timeout from a brownout from a genuine crash, and that one field will save weeks.
• Make the device report its own health before it fails, not after. A slow degradation in supply voltage or a rising internal temperature is a maintenance visit; the same device dead is a truck roll and an outage.
• Version everything and know what is deployed where. Across a mixed estate, "it works on the ones we tested" is not a defence.

The cost asymmetry is brutal. Diagnosing a fault on the bench is an hour. Diagnosing the same fault across a deployed estate, without instrumentation, through an intermittent that only occurs below five degrees, is a project.

Where interoperability multiplies the problem

Harsh-environment estates are rarely single-vendor. A roadside or trackside installation may combine cameras, encoders, sensors and controllers from several manufacturers, each interpreting the same specification slightly differently, all sharing a network nobody fully controls.

That is where environmental fragility and protocol fragility compound. A camera that renegotiates its link after a cold morning is an environmental problem. A VMS that never recovers the stream afterwards is an interoperability problem. The site sees one fault, and two suppliers each correctly report that their equipment is working.

Resolving that requires diagnosis at the wire rather than by substitution — capturing what actually happened on the network at the moment of failure, and attributing it to a device or a platform with evidence rather than opinion.

What this means for a specification

If you are writing a requirement for equipment that will be deployed outdoors, trackside, or on a factory floor, the environmental section is usually the weakest part of the document. Three additions do more than another ten degrees of temperature rating:

1. Specify thermal cycling, not just range. Number of cycles, rate of change, and the expected service life in years.
2. Require diagnostic instrumentation as a deliverable. Reset cause, supply voltage, internal temperature, and a log that survives power loss. Make it testable.
3. Define behaviour on degraded power, not only on power loss — including recovery from a supply that sags rather than one that disappears.

None of these are expensive at design time. All of them are expensive to add to a deployed estate.

How DevSpark helps

We work on embedded systems that live outdoors and in industrial environments — board bring-up and embedded Linux platforms, protocol-level fault diagnosis on mixed-vendor networks, and the instrumentation that makes a deployed estate diagnosable rather than mysterious.

Recent work includes re-engineering an OpenWrt-based video platform for multicast and ONVIF conformance across UK national road infrastructure, delivered under security clearance.

If you have equipment failing in the field and the pattern is not yet clear, that is the conversation we are most useful in — usually starting with a fixed-price diagnosis rather than a proposal.

---

This article is general engineering guidance, not a substitute for environmental qualification testing against the standards applicable to your sector.