2024d_execution_model/sections/OurModel.tex

\section{Detailed Intermittent Execution Model}
\label{sec:detailed_execution_model}

In this section, we describe our execution model and its implications for software design.
Sec.~\ref{sec:system_description} introduces the target architecture and the reference system used for evaluations.
In Sec.~\ref{sec:execution_model}, we present the proposed execution model, derived from key observations obtained through experimental results.
In the following three sections, we discuss how this model affects both the power efficiency and correctness of software design.
Finally, in Sec.~\ref{sec:other_architectures}, we evaluate the effectiveness of our model across systems with different architectural configurations. 

\subsection{System Description}
\label{sec:system_description}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/cropped/system.pdf}
    \caption{A typical hardware setup of intermittent systems.}
    \label{fig:hardware_setup}
\end{figure}

A typical intermittent system consists of two main components: a power management system and a computing system, as illustrated in Fig.~\ref{fig:hardware_setup}.
The power management system is responsible for accumulating the incoming energy into storage and providing a stable-voltage current to the computing system.
The computing system equips NVMs along with the MCU and peripherals, and utilize the NVMs for state retention between power failures.

This setup includes two notable decoupling capacitors that affect the execution model of intermittent systems.
The first one (C1 in the figure) is located within the power management system as voltage regulators require a capacitor larger than the device-specific minimum to ensure stable operation.
The second capacitor (C2) is part of the computing system and is used for stabilizing the operating voltage against sudden current draw.

Recent studies have increasingly explored 32-bit architectures for computing systems~\cite{shihIntermittent2024,wuIntOS2024,kimRapid2024,akhunovEnabling2023,kimLACT2024,kimLivenessAware2023,parkEnergyHarvestingAware2023,kortbeekWARio2022,khanDaCapo2023,barjamiIntermittent2024,songTaDA2024}, as emerging applications on intermittent systems, such as Deep Neural Networks (DNNs)~\cite{houTale2024,yenKeep2023,khanDaCapo2023,gobieskiIntelligence2019,islamEnabling2022,kangMore2022,leeNeuro2019,islamZygarde2020,custodeFastInf2024,barjamiIntermittent2024,songTaDA2024}, demand greater computational capabilities~\cite{bakarProtean2023a,carontiFinegrained2023}.
In this context, we employ a custom-built board featuring a 32-bit ARM Cortex-M33 processor (STM32L5, operating at 16Mhz) with 512KB of Ferroelectric RAM (FRAM) as our reference system.
For the power management system, we use a TI BQ25570-based board with power-on and power-off thresholds of 4.9 V and 3.4 V, respectively.
% A TI BQ25570 based board is used for the power management system, with power-on and off thresholds of 4.9V and 3.4V, respectively.
We empirically select 22uF and 220uF capacitors for C1 and C2, respectively, as these are the minimum capacitor sizes for stable checkpoint and recovery.
Sec.~\ref{sec:other_architectures} evaluates the generality of our model across different architectures, such as systems with Magnetic RAM (MRAM) and a 16-bit core (e.g., MSP430).

% In this work, our goal is to model the buffering effects of these capacitors and evaluate their implications on software designs.
% (Recent studies present the need for better computing capability~\cite{bakarProtean2023a})
% For model validation and evaluation, we use a custom-built board equipped with an ARM Cortex-M33 core and 512KB of FRAM.
% Our setup requires XXuF and 220uF capacitors for C1 and C2, respectively, for stable execution of checkpoint and recovery.
% Sec.~\ref{sec:other_architectures} evaluates our model in different architectures.

\subsection{Execution Model}
\label{sec:execution_model}

\begin{figure}
    \centering
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_8a_cropped.pdf}
        \caption{Voltage traces for one power cycle.}
        \label{fig:execution_trace_one_cycle}
        \vspace{5pt}
    \end{subfigure}
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_8b_cropped.pdf}
        \caption{Voltage traces around the first power-on.}
        \label{fig:execution_trace_detailed}
    \end{subfigure}
    \caption{Voltages trace of energy storage and Vdd.}
    \label{fig:execution_trace}
\end{figure}

To derive general execution model with the effects of decoupling capacitors, we first present a sample measurement from our reference system.
To achieve an operation time of 50ms under 1.5mA current supply, we use a 470uF capacitor for energy storage.
Fig.~\ref{fig:execution_trace_one_cycle} illustrates the voltage traces of the energy storage and the MCU operating voltage (Vdd) over a single power cycle.
Note that Vdd is maintained by decoupling capacitors after current supply from the power management system stops.
The shaded areas represent the periods that system executes the application code.
% Fig.~\ref{fig:execution_trace_one_cycle} shows the trace during one power cycle, and Fig.~\ref{fig:execution_trace_detailed} presents the first execution cycle in more detail.

Fig.~\ref{fig:execution_trace_detailed} presents the first execution cycle in more detail. It shows several interesting differences between the traditional execution model and the actual operation.
Among them, we highlight three key observations that affect software design decisions.

\begin{itemize}
    \item \textbf{O1}: The capacitor voltage drops rapidly to charge decoupling capacitor when the system wakes up ($t1$--$t2$).
    \item \textbf{O2}: The system operates at sub-voltage using the decoupling capacitors, even after power supply stops ($t4$--$t5$).
    \item \textbf{O3}: Decoupling capacitors discharge while the system is powered off (after $t5$, as shown in Fig.~\ref{fig:execution_trace_one_cycle}).
\end{itemize}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/cropped/detailed_execution_model.pdf}
    \caption{Detailed execution model of intermittent systems.}
    \label{fig:detailed_execution_model}
\end{figure}

% As we discuss in the following sections, all three observations significantly impact the performance of intermittent system designs.
% We propose a detailed execution model which reflects these observations.
Fig.~\ref{fig:detailed_execution_model} illustrates our detailed execution model, incorporating these key observations.
When the capacitor voltage reaches the power-on threshold, the voltage experience a rapid drop due to the buffering effects (\circled{1}), instead of gradual decline.
After initialization (\circled{2}), the system begins execution at normal operating voltage (\circled{3}), 3.3V for example.
When the voltage hits the power-off threshold, the power supply stops but system now starts to operate using the buffered energy (\circled{4}).
Since the voltage of the decoupling capacitor decreases as it discharges, the system executes at sub-normal voltage until it reaches the voltage it cannot operate (e.g., 2.5V).
% This voltage is known as Brown-Out Reset (BOR) voltage and is typically in a range of 1.7V to 2.5V in modern MCUs~\cite{}.
Finally, until the next power-on event, the remaining energy in decoupling capacitors continues to discharge (\circled{5}).

When designing intermittent systems, particularly those utilizing small capacitors, it is important for software designers to have clear understanding of this model.
In the following sections, we discuss the impact of our model to software design in more detail.

\subsection{Impact on Power Efficiency}
\label{sec:power_efficiency}

The traditional model implies that the energy consumed between power-on and power-off thresholds are entirely used in the computing system.
However, our model reveals that considerable energy is used for charging the decoupling capacitors (\textbf{O1}) and dissipated during power-off durations (\textbf{O3}).
This indicates that much smaller energy may be used for the useful computation compared to the designer's expectation.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/plot_expr_5_cropped.pdf}
    \caption{Distribution of energy consumed in a power cycle in different capacitor sizes (1mA current supply).}
    \label{fig:power_distribution}
\end{figure}

Fig.~\ref{fig:power_distribution} shows the distribution of the energy consumption for each stage of operation within one power cycle, averaged over 50 executions, where 1mA of input current is provided at 1.9V.
The x-axis represents capacitor sizes and the line in the secondary axis represents the average operation times for application code.
The checkpoint is executed by the interrupt from the power management system~\cite{}, which is generated when the capacitor voltage reaches the power-off threshold (3.4V).
Note that this is the most efficient point for checkpoint execution according to the traditional model.

The results shows that significant energy is wasted in the decoupling capacitors.
For example, 60.7\% of power is wasted during the power-off duration (denoted as \emph{Dischrged}) in 470uF case.
The discharging behavior can be modeled as an RC-discharging circuit (i.e., $q=CVe^{-\frac{1}{RC}t}$), which exhibits an exponential discharge rate.
As a result, the energy loss due to discharging is more expensive when the capacitor size is small.
In our case, 50\% of energy is discharged within the first 161 ms.
While the discharge rate decreases with larger capacitor sizes, it remains significant;
for example, in the 1320uF case, 28.5\% of energy is discharged, which is still non-negligible.
% The discharge rate decreases as the capacitor size increases, down to 28.5\% in 1320uF case, which is still not negligible.
% The cost is more expensive when the capacitor size is small since the discharge rate follows the RC-discharging circuits.
% While using smaller capacitors shortens the power-off durations, the discharging behavior penalizes them most since RC-discharging circuits discharge exponentially (in our case, 50\% of energy is discharged at the first 161 ms).
% As a result, 60.7\% of power is wasted in 470uF, and the rate decreases as the capacitor size increases, down to 28.5\% in 1320uF case.

Another important observation is the error introduced by the traditional model.
The traditional model expects both the energies, \emph{Execution} and \emph{Discharged}, are used for computation.
This introduces significant errors, up to 5.62x in 470uF setup.
In the same context, the traditional model predicts that using a 470uF capacitor instead of a 1320uF would result in only 1.22x overhead in energy efficiency, while the actual difference is 4.71x.
% However, our model shows that the actual energy efficiency differs by xx\% in reality, brining xx\% error in the traditional model.
This can significantly mislead system designers when they select capacitor sizes by considering tradeoffs between overall efficiency and reactiveness.
In Sec.~\ref{sec:design_guidelines}, we explore strategies to minimize overhead caused by discharging when designing software techniques.

% More importantly, this wasted energy is expected to be used for the computation in traditional execution model, as all the energy except for the initialization and checkpoint/recovery is expected to be used in computations.
% It brings significant errors between the two models in available energy for the execution.
% In 470uF case, the actual energy efficiency (Execution) and the expectation from the traditional model (Execution and Discharged) differs by 4.99 times.

% (Limitations of power failure injection and simulation based evaluations).

% In Sec.~\ref{sec:design_guidelines}, we discuss our guidelines to maximize power efficiency with software-level designs.

\subsection{Impact on Predicting Power Failures}
\label{sec:predicting_power_failures}

According to the traditional model, system states should be saved to NVM before reaching power-off threshold, as the system halts at this point.
On the other hand, our model shows that the system may continue operating using the energy stored in the decoupling capacitors (\textbf{O2}). 
Since modern MCUs can operate across a wide range of supply voltages (e.g., from 1.7V to 3.6V in STM32L5 and MSP430), the computing system is executed until the voltage of decoupling capacitors drops to the minimum operating level.
% Modern MCUs can operate on a range of supply voltages (e.g., from 1.7V to 3.6V for STM32L5 and MSP430).
% Since the voltage of decoupling capacitors decreases as the discharge, the computing system is executed until the voltage reaches the minimum operating voltage.
% While the voltage of decoupling capacitors decreases as they discharge, the computing system operates since modern MCUs can operate on a range of supply voltages (e.g., from 1.7V to 3.6V for STM32L5 and MSP430).
This makes the energy storage voltage not a reliable estimate of the remaining execution time.

\begin{figure}
    \centering
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_6a_cropped.pdf}
        \caption{Input current = 1mA.}
        \label{fig:sub_voltage_execution_1mA}
        \vspace{5pt}
    \end{subfigure}
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_6b_cropped.pdf}
        \caption{Input current = 3mA.}
        \label{fig:sub_voltage_execution_3mA}
    \end{subfigure}
    \caption{Ratio of sub-voltage operations in total execution time.}
    \label{fig:sub_voltage_execution}
\end{figure}

% Modern MCUs can operate on wide range of operating voltages (e.g., from 1.7V to 3.6V for STM32L5 and MSP430).

Fig.~\ref{fig:sub_voltage_execution} presents the ratio of the times executed under sub-voltage to the total execution times, averaged over 30 measurements.
The x-axis represents different capacitor sizes and the colors indicate the voltage levels at which the system stops operation.
We evaluate a range of stop voltages from 1.7V to 2.5V since not all components in the computing system may function at the lowest voltage (Sec.~\ref{sec:sub_normal_execution}).
Also, we examine two cases with different input currents of 1mA (Fig.~\ref{fig:sub_voltage_execution_1mA}) and 3mA (Fig.~\ref{fig:sub_voltage_execution_3mA}), to assess the impact of varying input power.

The figure shows that a significant portion of MCU operation occurs at sub-normal voltage.
For example, when 470uF capacitor is used at 1mA input current (Fig.~\ref{fig:sub_voltage_execution_1mA}), 82.8\% of computation takes place \emph{after} the power-off threshold.
This ratio decreases as the system powers off earlier (reducing sub-voltage operation time) or the input current increases (extending operation time at normal voltage).
Under 1000uF is the major focus of this paper.

These values can be directly translated to the inefficiencies of the system based on the traditional model.
For example, in the case of 470uF with 1mA input current, systems executing checkpoint at power-off threshold may operate 16.3ms.
However, the system could operate for an additional 29.4ms if the checkpoint is executed at 2.5V.
At the next power-on, the decoupling capacitors discharge to similar voltage levels in both cases, as their discharge behavior follows an exponential curve (Sec.~\ref{sec:power_efficiency}).
As a result, failing to utilize the available energy at sub-normal voltage introduces significant power efficiency overhead.
% Although early checkpoint execution may save some energy in decoupling capacitors, the saved energy is not preserved as discussed in Sec.~\ref{sec:power_efficiency}.
In Sec.~\ref{sec:design_guidelines}, we validate this aspect and propose methods to predict the power-off time more accurately.

\subsection{Impact of Sub-normal Voltage Execution}
\label{sec:sub_normal_execution}

The traditional model leads the software designers to assume that the system is executed under a stable voltage.
However, a significant portion of execution may happen after the power-off threshold at sub-normal voltage (\textbf{O3}), as discussed in Sec.~\ref{sec:predicting_power_failures}.
Being aware of this is crucial to software designers since the peripherals and analog components may function differently at sub-normal voltage.

Two of the most relevant examples are Analog-Digital Converters (ADCs) and external NVMs.
They play an important role in checkpointing, since ADCs are often used to estimate power-off time by reading the capacitor voltage and NVM serves as the storage for checkpoints.
At the same time, they are likely to operate at sub-normal voltages, as it is most efficient to execute checkpoint just before power-off.
% Incorrect execution of these components may lead to unsafe or incomplete checkpoint executions.

\begin{figure}
    \centering
    \begin{subfigure}{0.45\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_2_cropped.pdf}
        \caption{Analog-Digital Converter.}
        \label{fig:adc_error}
    \end{subfigure}
    \hfill
    \begin{subfigure}{0.52\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_3_cropped.pdf}
        \caption{External FRAM.}
        \label{fig:fram_drror}
    \end{subfigure}
    \caption{Incorrectly functioning components at sub-normal voltage.} 
    \label{fig:adc_and_fram_error}
\end{figure}

Fig.~\ref{fig:adc_error} shows the behavior of ADCs in sub-normal voltage.
ADC quantizes the input analog voltage into the range of discrete $2^n$ values from 0 to $V_{ref}$, where $n$ is a resolution and $V_{ref}$ is a reference voltage, and cannot read the input voltage larger than $V_{ref}$.  
As STM32L5 is designed to use Vdd as reference voltage, accessing the ADC during sub-normal voltage operation leads to inconsistent results.
As shown in the figure, the ADC returns values higher than the measurements since ADC representation range is decreased as Vdd drops.
As a result, during sub-normal voltage operation, the system may incorrectly interpret ADC results as there is sufficient energy and decide not to execute a checkpoint.

Also, intermittent systems typically designed with the use of peripherals, including sensors~\cite{yildizAdaptable2024,dangIoTree2022,afanasovBatteryless2020,maengAdaptive2020}, wireless communication modules~\cite{katanbafMultiScatter2021,dewinkelIntermittentlypowered2022,babatundeGreentooth2024} or external NVMs~\cite{dewinkelIntermittentlypowered2022,kimLACT2024,kimLivenessAware2023,akhunovEnabling2023}, which have their own minimum operating voltage requirements.
% Also, some peripherals may not work below certain voltage.
Fig.~\ref{fig:fram_drror} illustrates the error rate of FRAM in the reference system at different voltages, showing FRAM cannot operate reliably below 2.4V.
Since the system continues operating until it reaches the lowest MCU operation voltage (e.g., 1.7V), software designers must ensure that peripherals are accessed only at safe voltage levels.
Failing to this can result in corrupted data or incomplete checkpointing.
In Sec.~\ref{sec:design_guidelines}, we propose two techniques that can safely estimate the power-off time under sub-normal voltage conditions.

\subsection{Sensitivity to Architectural Designs}
\label{sec:other_architectures}

% Please add the following required packages to your document preamble:
% \usepackage{booktabs}
% \usepackage{multirow}
% \usepackage{graphicx}
\begin{table}[]
    \centering
    \caption{Architectures for generality evaluation}
    \label{tab:architectures}
    \renewcommand{\arraystretch}{0.9} % Reduce vertical spacing
    \setlength{\tabcolsep}{3pt} % Reduce horizontal spacing
    \resizebox{0.95\columnwidth}{!}{%
    \begin{tabular}{@{}cccccccc@{}}
    \toprule
    \multirow{2}{*}{} & \multirow{2.5}{*}{Core} & \multirow{2.5}{*}{\begin{tabular}[c]{@{}c@{}}Core\\ Freq.\end{tabular}} & \multicolumn{3}{c}{Capacitance (uF)} & \multirow{2.5}{*}{Current} & \multirow{2.5}{*}{Memory}                                   \\ \cmidrule(lr){4-6}
                      &                       &                                                                       & C1       & C2        & Storage       &                          &                                                           \\ \midrule
    A1              & STM32L5               & 16MHz                                                                 & 22       & 220       & 1,320         & 3mA                      & \begin{tabular}[c]{@{}c@{}}MRAM\\ (off-chip)\end{tabular} \\
    A2            & MSP430FR5994          & 8MHz                                                                  & 22       & 10        & 40            & 100uA                    & \begin{tabular}[c]{@{}c@{}}FRAM\\ (on-chip)\end{tabular}  \\ \bottomrule
    \end{tabular}%
    }
    \end{table}

To verify generality of our model, we evaluate it using two additional architectural setups.
Table~\ref{tab:architectures} shows the detailed parameters of the target architectures.
A1 shares the same configuration as the reference system but equips MRAM (Everspin MR5A16ACYS35) instead of FRAM.
This setup is included since MRAM is also gaining attention as a next generation NVM~\cite{akhunovEnabling2023,bakarProtean2023a,dewinkelIntermittentlypowered2022,wuIntOS2024}.
Second target is the MSP430, which has been the mostly adopted 16-bit platform in intermittent system research.
For both systems, the architectural parameters are set to achieve an operation time of approximately 50 ms.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/plot_expr_9_cropped.pdf}
    \caption{Energy breakdown and the ratio of sub-voltage operations in different architectures.}
    \label{fig:other_architectures}
\end{figure}

Fig.~\ref{fig:other_architectures} shows the results for different power-off voltages.
The bars on the left illustrate the energy breakdown in a single power cycle, and the bars on the right represent the ratio of the execution time operated at sub-voltage.
The most noticeable difference is ratio of energy consumed during the ramp-up and init stage.
While A1 consumes 63.4\% power at this stage on average, only 5.6\% of energy is consumed in A2.
This is because A1 is configured to use external MRAM, which exhibits significantly higher leakage current than FRAM used in the reference system.
In contrast, MSP430 (A2) is equipped with on-chip FRAM, which has much lower leakage.

Despite these differences, both architectures exhibit high sub-voltage execution rates, up to 55.5\% in A1 and 70.1\% in A2.
In addition, discharged energy takes considerable portion in both A1 (31.4\%) and A2 (52.0\%) at 3.3V power-off voltage configuration, which represents the techniques based on the traditional model that halt immediately at power-off threshold.
In summary, the evaluation demonstrates that the modeled buffering effects are general and their impacts are significant across different system architectures.
% In summary, the evaluation reveals that the buffering effect of system's capacitance and its implications are general in other systems.