Momentrack — Real-Time Tracking of Emergency-Response Resources for Canada’s Department of National Defence

In 2019, my company Momentaj won a federal R&D contract from Canada’s Department of National Defence (DND) through Innovative Solutions Canada. It was meant to answer a question first responders had been asking for years. During a major incident, police, fire and paramedics from several agencies work the same scene. How does anyone know where every truck, kit and responder is, and whether each one is available?

I led the project and designed the solution. In six months, a team of nine built Momentrack. It had low-power tags, a vehicle gateway, and a private cloud that put every tracked resource on a live incident map. We delivered every milestone on schedule and moved the concept from technology readiness level (TRL) 3 to TRL 4.

The challenge

DND’s Centre for Security Science runs the Canadian Safety and Security Program with Public Safety Canada, and its users are civilian first responders. In December 2018 it posted a challenge through ISC. It wanted real-time decision support for the people in command during critical incidents, when municipal, provincial and federal agencies, including the RCMP and DND, work together.

At the time, those teams tracked their resources with everything from spreadsheets to specialized software, and nothing gave them one shared picture. The challenge cited Project Responder 5, a US Department of Homeland Security study. In that study, responders ranked “the ability to know, in real time, the availability, location and status of all resources” among their most important capability needs. Canadian responders had described the same gap.

DND set six must-haves. A solution had to:

• let responders request resources from the field and track each request
• keep an inventory of what front-line vehicles carry
• build one repository of every resource available for an incident
• show critical resources on an incident map
• use standardized icons
• support GIS coordinates and layered, incident-specific maps

It also listed twelve desirable outcomes. They included resource details on demand, user queries and filters, real-time request status, open data formats, offline access to the last-known data, privacy, and pulling in data from every agency on scene.

What made it hard

• No single radio fits every resource. A fire extinguisher or a first-aid kit needs a tag that’s small, cheap and runs for months on a battery, which rules out a cellular modem. A fire truck has power to spare and has to reach the cloud from anywhere. Responders already carry phones.
• The newest network was the least proven. LTE-M, the low-power cellular standard built for devices like these, had only just launched in Canada. Modules were new and documentation was thin.
• The radios had never been combined this way. Bluetooth Low Energy (BLE), LoRa and LTE-M were each mature on their own. Running all three together, with a gateway bridging between them, was the untested part.
• The picture had to stay live. A command post needs positions that update in seconds, not a report after the fact. Data had to keep flowing even if part of the system failed.
• The data is sensitive. Where responders and their equipment are during an incident isn’t something to put on a public cloud by default.

How we approached it

Match the radio to the resource. Instead of forcing one technology onto everything, we defined four device types. Each one matched what a resource can carry and how far its signal needs to travel.

Research first, build second. We followed the V-model: design the concept, then prove each part against it. We ranked every requirement by how ready its technology already was. Anything we knew would work, such as assigning tags, printing labels or building reports, stayed out of scope. The six months went to the uncertain parts: radio interoperability, LTE-M on Canadian networks, and real-time visualization on a map.

Own the whole stack. The cloud ran on servers we operated ourselves, as stand-alone software with no dependency on AWS, Azure or Google Cloud. For the proof of concept we made one exception, Google Maps, and planned to replace it (see What came next).

Measure every hop. Every tag and gateway logged what it sent and received to an SD card. We compared those logs with what reached the database. That gave us a reliability figure for each link, not just for the system as a whole, and it showed exactly where data was being lost.

What we built

A data model for multi-agency incidents

Every agency on scene is a stakeholder. Each stakeholder owns resources of three kinds: assets (equipment, supplies, tools), people, and vehicles. A resource is tracked by a device attached to it or assigned to it. Devices are kept separate from resources so a tag can be put into inventory and later moved to a different resource. Each device carries sensors. Commanders draw incident zones on the map as polygons, and requests move resources into them.

Each resource type gets the devices that fit it:

• Assets: a Local Tag or a Direct Tag
• People: a Powered Tag or a Direct Tag
• Vehicles: a Gateway, a Powered Tag or a Direct Tag

The devices

For a proof of concept, off-the-shelf hardware was the fastest way to test radios, not circuit boards. We built every device except the phone on Pycom’s FiPy, a single module with BLE, LoRa and LTE-M radios. Sensor shields went on top, and the firmware was written in MicroPython.

A Local Tag has no GPS by design. It stays small and cheap, and its position is the position of the gateway or phone that hears it. That’s less precise, but good enough to say which vehicle a defibrillator is in.

The radio layer

• BLE: each Local Tag advertised its readings once a second in a custom packet modeled on Eddystone-TLM. The tag ran in peripheral-only mode to save power.
• LoRa: raw LoRa rather than LoRaWAN, on 927 MHz in the US902–928 band used in Canada, at +20 dBm with 500 kHz bandwidth. Tags only transmitted, never listened, which saved power and kept the channel free.
• LTE-M: we tested SIM cards from six providers: Telus, Rogers, Bell, Twilio, Hologram and iBasis. In early 2020, only Hologram, running on Rogers’ network, connected at all.
• Payloads: a Local Tag’s full set of readings fit in 19 bytes over LoRa, and in a single BLE advertisement. Gateways and Direct Tags sent JSON to the cloud.

The private cloud

The cloud had seven services, each on its own virtual machine on a VMware server we ran ourselves. Each one could fail or be updated without taking the others down. If the API stopped, the IoT Hub kept accepting data from the field and the queue held it until the API was back.

• IoT Hub: the front door for every device. It took HTTP(S) and MQTT(S), checked each device’s credentials and security settings, and put the message on the queue.
• Processor: decoded each payload into typed readings (sensor, value, unit and prefix).
• Data Access Layer: the only service that wrote to the database. It used bulk writes to keep up with many small readings arriving at once.
• Queue: RabbitMQ between every layer, with a topic for each kind of message.
• Repository: MongoDB as a replica set of two data nodes and an arbiter, with geospatial queries for incident zones.
• API: REST for queries and management, and SignalR to push live updates to browsers.
• Web app: a single-page Vue application.

The live incident map

The web app is where it all came together. Users could:

• define stakeholders, resources, devices and sensors
• draw incident zones as polygons on the map
• watch every resource move live, with standardized icons for trucks, ambulances, helicopters and field camps, map layers, and movement trails
• click any resource to see its owner, status and latest sensor readings
• browse each gateway’s and device’s raw data as it arrived, which is how we debugged the hardware
• chart and filter sensor history, and export any report to CSV
• create and track resource requests (basic management in the proof of concept)

To show a busy incident, we built a tag data simulator. It replayed an hour of real recorded device traffic, mixed with synthetic data, to the cloud every three seconds.

The request flow, as designed for the full system:

Proving it: ten tests

Every test compared what the devices logged locally with what arrived in the cloud.

What the numbers told us

• LoRa was the dependable short-range link. 95% of LoRa readings made it from tag to database.
• BLE was the weak link, and per-hop logging showed where it failed. Only 3,846 of 6,480 BLE advertisements reached the gateway. Of those that did, 98% reached the database. Almost all the loss happened in the air between tag and gateway, so the fix belonged in the radio design, not the cloud.
• LTE-M worked, but the market was young. 7.5 seconds to connect and 17 kbps are plenty for location updates. Only one SIM provider out of six connected, though.
• GPS cold starts are slow. Two minutes on average to get a first fix meant a production tag would have to keep its GPS warm, or take fixes only when it moves.
• One device misbehaved. One of the four Direct Tags was unstable in every road-test run. We sent its logs to the manufacturer to investigate.

My role

• Led the bid. I led Momentaj’s proposal. DND’s evaluators gave it full marks for innovation, citing three significant improvements: multi-range radio for a reliable, highly available network; a dynamic dashboard; and battery management for low-power tags.
• Designed the solution end to end. I owned the solution architecture across devices, radios and the cloud: the four device types, the choice of which radio to use where, and the layered, queue-based cloud.
• Wrote code across the stack. I was hands-on in the device firmware, the IoT Hub, the Processor and Data Access Layer, and the API and web app.
• Ran the project. I managed the team of nine through five milestones and reported to DND’s technical authority at Defence Research and Development Canada. The team included a hardware lead and radio designer, a senior .NET developer, a front-end developer, a UX/UI designer, a business analyst, and three junior hardware and firmware engineers, one of them a new graduate and one a University of Waterloo co-op student.
• Validated the results. I analyzed the test logs and validated the results myself.
• Led the Phase 2 proposal. I shaped the architecture and the plan for the Phase 2 proposal described below.

The cloud side reused the architecture I had already proven on uBeac, my IoT platform: ingest fast, queue everything, decode on the server, store, and stream live to the browser. That’s why a small team could have a seven-service private cloud running by the second milestone.

Outcome

• TRL 3 → TRL 4 at both system and subsystem level
• Five milestones out of five delivered on schedule, through the first COVID-19 lockdown. The final report went to DND on May 24, 2020.
• A working end-to-end system: four device types, three radio technologies, seven cloud services and a live incident map
• Measured baselines for the next phase: LoRa 95%, gateway to cloud 100%, LTE-M 7.5 s to connect at 17 kbps, 98.7% of road-test location reports delivered
• Invited by Canada to propose Phase 2

What came next

In December 2020, Canada invited Momentaj to submit a Phase 2 proposal. Phase 2 offered up to two years and $1 million to turn the proof of concept into a working prototype that DND would own. We proposed an 18-month plan aimed at TRL 8:

• Start with the responders. The plan began with five months of requirements work with the people who would use the system: interviews, workflow analysis and process models, all before any hardware design. This answered the main gap DND’s evaluators had found in our Phase 1 plan.
• Real hardware. Custom tags and gateways, designed with Canadian electronics design firms. The plan called for three rounds of engineering prototypes, each tested against the specification and integrated with the cloud, plus enclosures and certification. Local Tags would run for months on coin cells, waking when they moved or on a timer.
• Mobile apps. iOS and Android apps would let responders see nearby resources, send and acknowledge requests, and act as Powered Tags.
• A cloud built for scale. Storage would be split in three: live data that expires automatically, an archive tuned for queries over time, and business records. Other planned pieces:
  • real-time streaming as its own service, with compression
  • encrypted device payloads
  • an integration agent to pull resource data from other agencies’ files, databases and systems
  • single sign-on with Active Directory and LDAP
  • English and French
  • drag-and-drop dashboards
• Maps that run anywhere. Open-source mapping (Leaflet, OpenLayers and OpenStreetMap) hosted on-premises would replace Google Maps, so the system could work offline or on a closed network.
• Two-way communication. Configuration and firmware updates would be sent over the air from the cloud to the devices.

The plan’s workstreams, with the time budgeted for each. They overlapped, so the whole plan ran 18 months:

• Initiation: 0.5 month
• Requirements with first responders: 5 months
• Design: 3 months
• Cloud application: 8 months
• iOS and Android apps: 4 months
• Hardware, in three engineering-prototype rounds: 10.5 months
• Test, verification and integration: 4 months
• Delivery, training and close: 2.5 months

DND didn’t go ahead with Phase 2 for this challenge, so Momentrack ended at the proof-of-concept stage.

What I learned

• Measure every hop, not just the whole system. An end-to-end number would have said “BLE is unreliable.” Logging at each hop showed that nearly all the loss was in the air between tag and gateway, which told us exactly what to fix.
• The newest link is the riskiest dependency. LTE-M was the right technology for the job, and in early 2020 one SIM provider out of six worked. When a design depends on a network that just launched, test the network before trusting the datasheet.
• Listen to the evaluators, even after you’ve won. DND’s reviewers pointed out two gaps in our Phase 1 plan. It didn’t spend enough time with the first responders who would use the system, and it didn’t name radio interoperability as a risk. Our weakest test result came from the multi-radio gateway, the area they had flagged. Our Phase 2 plan started with the users.
• Scope by uncertainty. Ranking every requirement by technology readiness let a team of nine prove the risky parts in six months, instead of spreading itself across features we already knew how to build.
• Decide early what you won’t depend on. Running our own cloud was a deliberate choice for sensitive data. Google Maps was the one shortcut we took, and the Phase 2 plan had to budget time to undo it.

[
  {
    "id": "1A:2A:3A:4A:5A:6A",
    "ts": 1542326605,
    "sensors": [
      { "id": "SignalStrength", "type": 17, "unit": 22, "prefix": 0,
        "data": { "rssi": -86, "txPower": -59 } },
      { "id": "Temperature", "type": 4, "unit": 2, "prefix": 0, "value": 26 }
    ]
  }
]

Stack

Pycom FiPy (BLE, LoRa, LTE-M) · Pysense and Pytrack shields · MicroPython · Android · C# · .NET Core / ASP.NET Core 2.2 · MQTTnet · RabbitMQ · MessagePack · MongoDB (replica set, geospatial queries) · IdentityServer · SignalR · Swagger · Serilog · Vue 2 · Vuex · Bootstrap-Vue · Highcharts · Google Maps · VMware ESXi · Azure DevOps

References

• ISC challenge page
• ISC awarded companies list