2024d_execution_model/sections/OurModel.tex

\section{Detailed Intermittent Execution Model}
\label{sec:detailed_execution_model}

In this section, we describe our execution model and its implications for software design.
Sec.~\ref{sec:system_description} introduces the target architecture and the reference system used for evaluations.
Sec.~\ref{sec:execution_model} presents our execution model, derived from key observations obtained through experimental results.
In the following three sections, we discuss how this model affects both the power efficiency and correctness of software design.
Finally, in Sec.~\ref{sec:other_architectures}, we evaluate the effectiveness of our model across systems with various architectural configurations. 

\subsection{Target Architecture and Reference System}
\label{sec:system_description}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/cropped/system.pdf}
    \caption{A typical hardware setup of intermittent systems.}
    \label{fig:hardware_setup}
\end{figure}

A typical intermittent system consists of two main components: a power management system and a computing system, as shown in Fig.~\ref{fig:hardware_setup}.
The power management system collects incoming energy into storage ($C_{ES}$) and supplies a stable-voltage current to the computing system.
% The power management system is responsible for accumulating the incoming energy into storage ($C_{ES}$) and providing a stable-voltage current to the computing system.
The computing system equips NVMs along with an MCU and peripherals, and utilize the NVMs for state retention between power failures.

This setup includes two notable decoupling capacitors that affect the execution model of intermittent systems.
% The first one (C1 in the figure) is located within the power management system, as voltage regulators require a capacitance above the device-specific minimum for stable operation.
The first one (C1) is located in the power management system, required by voltage regulators to ensure stable operation.
The second capacitor (C2) is part of the computing system and used to stabilize the operating voltage against sudden current draw.

Recent studies have increasingly explored 32-bit architectures for computing systems~\cite{shihIntermittent2024,wuIntOS2024,kimRapid2024,akhunovEnabling2023,kimLACT2024,kimLivenessAware2023,parkEnergyHarvestingAware2023,kortbeekWARio2022,khanDaCapo2023,barjamiIntermittent2024,songTaDA2024}, as emerging applications on intermittent systems, such as Deep Neural Networks (DNNs)~\cite{houTale2024,yenKeep2023,khanDaCapo2023,gobieskiIntelligence2019,islamEnabling2022,kangMore2022,leeNeuro2019,islamZygarde2020,custodeFastInf2024,barjamiIntermittent2024,songTaDA2024}, demand greater computational capabilities~\cite{bakarProtean2023a,carontiFinegrained2023}.
In this context, we employ a custom-built board featuring a 32-bit ARM Cortex-M33 processor (STM32L5, operating at 16Mhz) with 512KB of Ferroelectric RAM (FRAM, Infineon FM22L16) as our reference system.
For the power management system, we use a TI BQ25570-based board configured with $V_h$ = 4.9V and $V_l$ = 3.4V.
% For the power management system, we use a TI BQ25570-based board with power-on and power-off thresholds of 4.9 V and 3.4 V, respectively.
% A TI BQ25570 based board is used for the power management system, with power-on and off thresholds of 4.9V and 3.4V, respectively.
We empirically select 22uF and 220uF capacitors for C1 and C2, respectively, as smaller capacitors fail to provide a reliable voltage for checkpoint and recovery.
% We empirically select 22uF and 220uF capacitors for C1 and C2, respectively, as these are the minimum capacitor sizes for stable checkpoint and recovery.
Sec.~\ref{sec:other_architectures} evaluates the generality of our model across different architectures, such as systems with different NVM (e.g., Magnetic RAM, MRAM) and a 16-bit core (e.g., MSP430).

% In this work, our goal is to model the buffering effects of these capacitors and evaluate their implications on software designs.
% (Recent studies present the need for better computing capability~\cite{bakarProtean2023a})
% For model validation and evaluation, we use a custom-built board equipped with an ARM Cortex-M33 core and 512KB of FRAM.
% Our setup requires XXuF and 220uF capacitors for C1 and C2, respectively, for stable execution of checkpoint and recovery.
% Sec.~\ref{sec:other_architectures} evaluates our model in different architectures.

\subsection{Execution Model}
\label{sec:execution_model}

\begin{figure}
    \centering
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_8a_cropped.pdf}
        \caption{Voltage traces for one power cycle.}
        \label{fig:execution_trace_one_cycle}
        \vspace{3pt}
    \end{subfigure}
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_8b_cropped.pdf}
        \caption{Voltage traces of the first execution cycle.}
        \label{fig:execution_trace_detailed}
    \end{subfigure}
    \caption{Voltages trace of energy storage ($V_{ES}$) and $V_{dd}$.}
    \label{fig:execution_trace}
\end{figure}

To derive a general execution model with the effects of decoupling capacitors, we first present a sample measurement from our reference system.
In this paper, we denote the voltage of the energy storage $C_{ES}$ as $V_{ES}$ and the MCU operating voltage as $V_{dd}$.
To achieve an operation time of 50 ms under 1.5mA current supply, we use a 470uF capacitor for $C_{ES}$.
Fig.~\ref{fig:execution_trace_one_cycle} illustrates the voltage traces of $V_{ES}$ and $V_{dd}$ over a single power cycle.
Note that $V_{dd}$ is maintained by decoupling capacitors once the power supply from the power management system stops.
The shaded areas represent the periods that system executes the application code.
% Fig.~\ref{fig:execution_trace_one_cycle} shows the trace during one power cycle, and Fig.~\ref{fig:execution_trace_detailed} presents the first execution cycle in more detail.

Fig.~\ref{fig:execution_trace_detailed} presents the first execution cycle in more detail. It reveals several differences between the traditional execution model and the actual operation.
Among them, we highlight three key observations that affect software design decisions:

\begin{itemize}
    \item \textbf{O1}: The capacitor voltage ($V_{ES}$) drops rapidly to charge decoupling capacitors when the system wakes up ($t1$--$t2$).
    \item \textbf{O2}: System operates at sub-normal voltage using decoupling capacitors, even after power supply stops ($t3$--$t4$).
    \item \textbf{O3}: Decoupling capacitors discharge while the system is powered off (after $t4$, as shown in Fig.~\ref{fig:execution_trace_one_cycle}).
\end{itemize}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/cropped/detailed_execution_model.pdf}
    \caption{Detailed execution model of intermittent systems.}
    \label{fig:detailed_execution_model}
\end{figure}

% As we discuss in the following sections, all three observations significantly impact the performance of intermittent system designs.
% We propose a detailed execution model which reflects these observations.
Fig.~\ref{fig:detailed_execution_model} illustrates our detailed execution model, incorporating these key observations.
When $V_{ES}$ reaches $V_h$, the voltage experience a rapid drop due to the buffering effects (\circled{1}), instead of gradual decline.
After initialization (\circled{2}), the system begins execution at normal operating voltage (\circled{3}), 3.3V for example.
When the voltage hits $V_l$, the power supply stops but system now starts to operate using the buffered energy (\circled{4}).
Since the voltage of the decoupling capacitors decreases as they discharge, the system executes at sub-normal voltage until it reaches the voltage level it cannot operate (e.g., 2.5V in Fig.~\ref{fig:execution_trace}).
% This voltage is known as Brown-Out Reset (BOR) voltage and is typically in a range of 1.7V to 2.5V in modern MCUs~\cite{}.
Finally, until the next power-on event, the remaining energy in decoupling capacitors continues to discharge (\circled{5}).

When designing intermittent systems, particularly those utilizing small capacitors, understanding the effects described by this model is critical.
% When designing intermittent systems, particularly those utilizing small capacitors, it is important for software designers to have clear understanding of this model.
In the following sections, we discuss impacts of our model to software design in more detail.

\subsection{Impact on Power Efficiency}
\label{sec:power_efficiency}

The traditional model implies that the energy consumed between $V_h$ and $V_l$ is entirely used in the computing system.
However, our model reveals that considerable energy is used for charging the decoupling capacitors (\textbf{O1}) and dissipated during power-off durations (\textbf{O3}).
This indicates that much smaller energy may be used for the useful computation compared to the designer's expectation.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/plot_expr_5_cropped.pdf}
    % \caption{Distribution of energy consumed in a power cycle in different capacitor sizes (1mA current supply).}
    \caption{Distribution of energy consumed in a power cycle.}
    \label{fig:power_distribution}
\end{figure}

Fig.~\ref{fig:power_distribution} shows the distribution of the energy consumption for each stage of operation within one power cycle, averaged over 50 executions, where 1mA of input current is provided at 1.9V.
The x-axis represents the size of $C_{ES}$ and the line in the secondary axis represents the average operation times for application code.
The checkpoint is executed by the interrupt from the power management system~\cite{jayakumarQUICKRECALL2014,maengSupporting2019,balsamoHibernus2016,balsamoHibernus2015,kortbeekTimesensitive2020}, which is generated when $V_{ES}$ reaches $V_l$ (3.4V).
Note that this is the most efficient point for checkpoint execution according to the traditional model (i.e., just before the poweroff).

The results shows that significant energy is wasted in the decoupling capacitors.
For example, in 470uF case, 60.7\% of the energy is lost during the power-off duration (denoted as \emph{Discharged}), leaving only 13.1\% of the energy for computation.
While the ratio of \emph{Discharged} decreases with larger $C_{ES}$, it remains substantial;
for example, in the 1320uF case, 28.5\% of energy is discharged, which is still non-negligible.

This is because the discharging behavior can be modeled as an RC-discharging circuit (i.e., $q=CVe^{-\frac{t}{RC}}$), which exhibits an exponential discharge rate.
Indeed, 50\% of the energy is discharged within the first 161 ms in our measurements.
Since recharging $C_{ES}$ takes 2.13 secs even in 470uF configuration, most of the buffered energy is lost before the next power-on, regardless of the size of $C_{ES}$.
As a result, the energy loss ratio due to discharging is larger with smaller capacitors.
% The discharge rate decreases as the capacitor size increases, down to 28.5\% in 1320uF case, which is still not negligible.
% The cost is more expensive when the capacitor size is small since the discharge rate follows the RC-discharging circuits.
% While using smaller capacitors shortens the power-off durations, the discharging behavior penalizes them most since RC-discharging circuits discharge exponentially (in our case, 50\% of energy is discharged at the first 161 ms).
% As a result, 60.7\% of power is wasted in 470uF, and the rate decreases as the capacitor size increases, down to 28.5\% in 1320uF case.

Another important observation is the error introduced by the traditional model.
The traditional model expects both the energies, \emph{Execution} and \emph{Discharged}, are used for computation.
This introduces huge errors, up to 5.62x in 470uF setup, for example.
In the same context, the traditional model predicts that using a 470uF capacitor for $C_{ES}$ instead of a 1320uF would result in only 1.22x overhead in energy efficiency, while the actual difference is 4.71x.
% However, our model shows that the actual energy efficiency differs by xx\% in reality, brining xx\% error in the traditional model.
This can significantly mislead system designers when they select capacitor sizes by considering tradeoffs between overall efficiency and reactiveness.
In Sec.~\ref{sec:design_guidelines}, we explore strategies to minimize the inefficiencies caused by discharging when designing software techniques.

% More importantly, this wasted energy is expected to be used for the computation in traditional execution model, as all the energy except for the initialization and checkpoint/recovery is expected to be used in computations.
% It brings significant errors between the two models in available energy for the execution.
% In 470uF case, the actual energy efficiency (Execution) and the expectation from the traditional model (Execution and Discharged) differs by 4.99 times.

% (Limitations of power failure injection and simulation based evaluations).

% In Sec.~\ref{sec:design_guidelines}, we discuss our guidelines to maximize power efficiency with software-level designs.

\subsection{Impact on Predicting Power Failures}
\label{sec:predicting_power_failures}

According to the traditional model, system states should be saved to NVM before $V_{ES}$ reaches $V_l$, as the system is expected to halt at this point.
On the other hand, our model shows that the system may continue operating using the energy stored in the decoupling capacitors (\textbf{O2}). 
Since modern MCUs can operate across a wide range of supply voltages (e.g., 1.7V to 3.6V in STM32L5 and MSP430), the computing system operates until the voltage of decoupling capacitors drops to the minimum operating level.
% Modern MCUs can operate on a range of supply voltages (e.g., from 1.7V to 3.6V for STM32L5 and MSP430).
% Since the voltage of decoupling capacitors decreases as the discharge, the computing system is executed until the voltage reaches the minimum operating voltage.
% While the voltage of decoupling capacitors decreases as they discharge, the computing system operates since modern MCUs can operate on a range of supply voltages (e.g., from 1.7V to 3.6V for STM32L5 and MSP430).
This makes $V_{ES}$ not a reliable indicator for the imminent power-off.
% This makes the $V_{ES}$energy storage voltage not a reliable estimate of the remaining execution time.

\begin{figure}
    \centering
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_6a_cropped.pdf}
        \caption{Input current = 1mA.}
        \label{fig:sub_voltage_execution_1mA}
        \vspace{3pt}
    \end{subfigure}
    \begin{subfigure}{\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_6b_cropped.pdf}
        \caption{Input current = 3mA.}
        \label{fig:sub_voltage_execution_3mA}
    \end{subfigure}
    \caption{Ratio of sub-voltage operations in total execution time.}
    \label{fig:sub_voltage_execution}
\end{figure}

% Modern MCUs can operate on wide range of operating voltages (e.g., from 1.7V to 3.6V for STM32L5 and MSP430).

Fig.~\ref{fig:sub_voltage_execution} presents the ratio of the times executed under sub-normal voltages to the total execution times, averaged over 30 measurements.
The x-axis represents the sizes of $C_{ES}$ and the colors indicate the voltage levels at which the system stops operation.
We evaluate a range of stop voltages from 1.7V to 2.5V since not all components in the computing system may function at the lowest voltage level (Sec.~\ref{sec:sub_normal_execution}).
Also, we examine two cases with input currents of 1mA (Fig.~\ref{fig:sub_voltage_execution_1mA}) and 3mA (Fig.~\ref{fig:sub_voltage_execution_3mA}), to assess the impact of input power.

The figure shows that a significant portion of MCU operation occurs at sub-normal voltages.
For example, when 470uF capacitor is used at 1mA input current (Fig.~\ref{fig:sub_voltage_execution_1mA}), 82.8\% of computation takes place \emph{after} the power-off threshold.
This ratio decreases as the system stops earlier (reducing sub-voltage operation time) or the input current increases (extending operation time at normal voltage).
However, at least 13.0\% of computations are operated at sub-normal voltages even in highly optimistic configurations (1320uF in Fig.~\ref{fig:sub_voltage_execution_3mA}).
% Overall, the average sub-voltage operation ratio is xx\% for the configurations exhibiting less than 100 ms, which is the main focus of this paper. 

These values can be directly translated to the inefficiencies of the systems based on the traditional model.
For example, in the case of 470uF with a input current of 1mA, systems executing checkpoint at $V_l$ may operate 16.3 ms.
However, the system could operate for an additional 29.4 ms if the checkpoint can be delayed until 2.5V.
At the next power-on, the decoupling capacitors discharge to similar voltage levels in both cases, as discussed in Sec.~\ref{sec:power_efficiency}.
As a result, failing to utilize the buffered energy at sub-normal voltages introduces significant power inefficiency.
% Although early checkpoint execution may save some energy in decoupling capacitors, the saved energy is not preserved as discussed in Sec.~\ref{sec:power_efficiency}.
In Sec.~\ref{sec:use_vdd_for_checkpoint}, we validate this aspect and propose methods to fully utilize the buffered energy.

\subsection{Impact of Sub-normal Voltage Execution}
\label{sec:sub_normal_execution}

The traditional model leads software designers to assume that the system is executed under a stable voltage.
However, a significant portion of execution may happen after the power-off threshold at sub-normal voltages (\textbf{O3}), as discussed in Sec.~\ref{sec:predicting_power_failures}.
Being aware of this is crucial to software designers since analog components and peripherals may function differently at sub-normal voltages.
Two relevant examples are Analog-Digital Converters (ADCs) and external NVMs.

ADCs are commonly used to determine when to execute a checkpoint by reading $V_{ES}$.
It quantizes the input analog voltage into discrete $2^n$ values, ranging from 0 to the given reference voltage, where $n$ is a resolution.
Using smaller reference voltage increases sensitivity of ADC at the cost of reduced representation range.
% They play an important role in checkpointing, since ADCs are commonly used to determine when to execute a checkpoint by reading $V_{ES}$, and NVM serves as the storage for the checkpoints.
% They play an important role in checkpointing: ADCs are commonly used to determine when to execute a checkpoint by reading $V_{ES}$ and NVM serves as the storage for the checkpoints.
% At the same time, they are likely to operate at sub-normal voltages, as checkpoint executions typically happen just before the power-off.
% Incorrect execution of these components may lead to unsafe or incomplete checkpoint executions.

\begin{figure}
    \centering
    \begin{subfigure}{0.45\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_2_cropped.pdf}
        \caption{Analog-Digital Converter.}
        \label{fig:adc_error}
    \end{subfigure}
    \hfill
    \begin{subfigure}{0.52\linewidth}
        \includegraphics[width=\textwidth]{figs/plot_expr_3_cropped.pdf}
        \caption{External FRAM.}
        \label{fig:fram_drror}
    \end{subfigure}
    \caption{Incorrect operations at sub-normal voltages.} 
    \label{fig:adc_and_fram_error}
\end{figure}

Fig.~\ref{fig:adc_error} shows the behavior of ADCs, where the execution in sub-normal voltages are depicted in gray.
% ADC quantizes the input analog voltage into discrete $2^n$ values, ranging from 0 to the given reference voltage, where $n$ is a resolution.
% Therefore, using smaller reference voltage increases sensitivity of ADC at the cost of reduced representation range.
% Since $n$ is fixed, using smaller reference voltage increases sensitivity of the ADC at the cost of reduced representation range.
Because STM32L5 uses $V_{dd}$ as a reference voltage, accessing the ADC under sub-normal voltages produces inconsistent results.
As shown in the figure, the ADC returns values higher than the actual measurements since its representation range decreases as $V_{dd}$ drops.
Consequently, ADC may mislead the system into overestimating the energy in $C_{ES}$ during sub-normal voltage executions, potentially leading to checkpoint miss and a loss of progress for the entire power cycle.
% This can result in failing to execute a checkpoint, leading to the loss of progress during the entire power cycle.
% As a result, during sub-normal voltage operation, the system may misinterpret ADC results as indicating sufficient energy in $C_{ES}$ and fail to execute a checkpoint, resulting in loss of progress during the power cycle.
% As a result, during sub-normal voltage operation, the system may inccor ADC results as indicating sufficient energy in $C_{ES}$ and decide not to execute a checkpoint, resulting in loss of the progress during the power cycle.

Also, intermittent systems typically designed to operate with peripherals such as sensors~\cite{yildizAdaptable2024,dangIoTree2022,afanasovBatteryless2020,maengAdaptive2020}, wireless communication modules~\cite{katanbafMultiScatter2021,dewinkelIntermittentlypowered2022,babatundeGreentooth2024} or external NVMs~\cite{dewinkelIntermittentlypowered2022,kimLACT2024,kimLivenessAware2023,akhunovEnabling2023}, which have their own minimum operating voltage requirements.
% Also, some peripherals may not work below certain voltage.
Fig.~\ref{fig:fram_drror} illustrates the error rate of FRAM in the reference system at different voltages, showing FRAM cannot operate reliably below 2.4V.
Since the system continues operating until it reaches the lowest MCU operation voltage (e.g., 1.7V), software designers must ensure that peripherals are accessed only at safe voltage levels.
Failing to do so can result in corrupted sensor data or unsafe checkpointing.
% In Sec.~\ref{sec:design_guidelines}, we propose two techniques that can safely estimate the power-off time under sub-normal voltage conditions.

\subsection{Sensitivity to Architectural Designs}
\label{sec:other_architectures}

% Please add the following required packages to your document preamble:
% \usepackage{booktabs}
% \usepackage{multirow}
% \usepackage{graphicx}
\begin{table}[]
    \centering
    \caption{Architectures for generality evaluation.}
    \label{tab:architectures}
    \renewcommand{\arraystretch}{0.9} % Reduce vertical spacing
    \setlength{\tabcolsep}{3pt} % Reduce horizontal spacing
    \resizebox{0.95\columnwidth}{!}{%
    \begin{tabular}{@{}cccccccc@{}}
    \toprule
    \multirow{2}{*}{} & \multirow{2.5}{*}{Core} & \multirow{2.5}{*}{\begin{tabular}[c]{@{}c@{}}Core\\ Freq.\end{tabular}} & \multicolumn{3}{c}{Capacitance (uF)} & \multirow{2.5}{*}{Current} & \multirow{2.5}{*}{Memory}                                   \\ \cmidrule(lr){4-6}
                      &                       &                                                                       & C1       & C2        & C\textsubscript{ES}       &                          &                                                           \\ \midrule
    A1              & STM32L5               & 16MHz                                                                 & 22       & 220       & 1,320         & 3mA                      & \begin{tabular}[c]{@{}c@{}}MRAM\\ (off-chip)\end{tabular} \\
    A2            & MSP430FR5994          & 8MHz                                                                  & 22       & 10        & 40            & 100uA                    & \begin{tabular}[c]{@{}c@{}}FRAM\\ (on-chip)\end{tabular}  \\ \bottomrule
    \end{tabular}%
    }
    \end{table}

To evaluate the generality of the proposed model, we assess it across two additional architectural setups.
Table~\ref{tab:architectures} shows the detailed parameters of the target architectures.
A1 shares the same configuration as the reference system but equips MRAM (Everspin MR5A16ACYS35), which is gaining attention as a next generation NVM~\cite{akhunovEnabling2023,bakarProtean2023a,dewinkelIntermittentlypowered2022,wuIntOS2024}, instead of FRAM.
% This setup is included since MRAM is also gaining attention as a next generation NVM~\cite{akhunovEnabling2023,bakarProtean2023a,dewinkelIntermittentlypowered2022,wuIntOS2024}.
Second target is MSP430 equipped with on-chip FRAM, a widely adopted 16-bit platform in intermittent system research.
For both systems, the architectural parameters are configured to achieve an operation time of approximately 50 ms.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figs/plot_expr_9_cropped.pdf}
    \caption{Energy breakdown and the ratio of sub-voltage operations in different architectures.}
    \label{fig:other_architectures}
\end{figure}

Fig.~\ref{fig:other_architectures} shows the results for different power-off voltages.
The bars on the left illustrate the energy breakdown in a single power cycle, and the bars on the right represent the ratio of the sub-normal voltage executions.
The most noticeable difference is ratio of energy consumed during the \emph{Ramp-up \& Init} stage.
While A1 consumes 63.4\% power at this stage on average, only 5.6\% of energy is consumed in A2.
This is because A1 is configured with an external MRAM, which exhibits significantly higher leakage current, even compared to the FRAM used in the reference system.
In contrast, A2 is equipped with on-chip FRAM, which has much lower leakage.

Despite these differences, both architectures exhibit high sub-normal voltage execution rates, up to 55.5\% in A1 and 70.1\% in A2.
In addition, discharged energy takes considerable portion in both A1 (31.4\%) and A2 (52.0\%) at 3.3V power-off voltage configuration, which represents the techniques based on the traditional model that halt immediately at $V_{ES}$.
In summary, the evaluation demonstrates that the modeled buffering effects are general and their impacts are significant across different system architectures.
% In summary, the evaluation reveals that the buffering effect of system's capacitance and its implications are general in other systems.