[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [PATCH v2] balloon: transitional device support
On Sun, Apr 26, 2015 at 08:40:10PM +0200, Michael S. Tsirkin wrote: > Virtio 1.0 cs02 doesn't include a modern balloon device. At some > point we'll likely define an incompatible interface with a > different ID and different semantics. But for now, it's not a > big effort to support a transitional balloon device: this has the > advantage of supporting existing drivers, transparently, as well > as transports that don't allow mixing virtio 0 and virtio 1 > devices. And balloon is an easy device to test, so it's also > useful for people to test virtio core handling of transitional > devices. > > Three issues with legacy hypervisors have been identified: > 1. Actual value is actually used, and is necessary for management > to work. Luckily 4 byte config space writes are now atomic. > When using old guests, hypervisors can detect access to the last byte. > When using old hypervisors, drivers can use atomic 4-byte accesses. > 2. Hypervisors actually didn't ignore the stats from the first > buffer supplied. This means the values there would be > incorrect until hypervisor resends the request. > Add a note suggesting hypervisors ignore the 1st buffer. > 3. QEMU simply over-writes stats from each buffer it gets. > Thus if driver supplies a different subset of stats > on each request, stale values will be there. > Require drivers to supply the same subset on each > request. This also gives us a simple way to figure out > which stats are supported. > > Please note the issues are there in legacy mode only: > we need to fix them even if we don't implement a modern > interface for balloon. > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > --- > > changes from v1: > dropped stats format change > added a bunch of normative statements > documented work-arounds for legacy hypervisor bugs > > conformance.tex | 24 +++++++- > content.tex | 173 ++++++++++++++++++++++++++++++++++++++++++++++++-------- > 2 files changed, 170 insertions(+), 27 deletions(-) > > diff --git a/conformance.tex b/conformance.tex > index 80b333f..c51287a 100644 > --- a/conformance.tex > +++ b/conformance.tex > @@ -15,7 +15,7 @@ Conformance targets: > \begin{itemize} > \item Clause \ref{sec:Conformance / Driver Conformance}, > \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance}. > - \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Console Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Entropy Driver Conformance} or \ref{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}. > + \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Console Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Entropy Driver Conformance} or \ref{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}. > \end{itemize} > \item[Device] A device MUST conform to three conformance clauses: > \begin{itemize} > @@ -123,6 +123,16 @@ An entropy driver MUST conform to the following normative statements: > \item \ref{drivernormative:Device Types / Entropy Device / Device Operation} > \end{itemize} > > +\subsection{Traditional Memory Balloon Driver Conformance}\label{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance} > + > +A traditional memory balloon driver MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{drivernormative:Device Types / Memory Balloon Device / Feature bits} > +\item \ref{drivernormative:Device Types / Memory Balloon Device / Device Operation} > +\item \ref{drivernormative:Device Types / Memory Balloon Device / Device Operation / Memory Statistics} > +\end{itemize} > + > \subsection{SCSI Host Driver Conformance}\label{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance} > > An SCSI host driver MUST conform to the following normative statements: > @@ -230,6 +240,16 @@ An entropy device MUST conform to the following normative statements: > \item \ref{devicenormative:Device Types / Entropy Device / Device Operation} > \end{itemize} > > +\subsection{Traditional Memory Balloon Device Conformance}\label{sec:Conformance / Device Conformance / Traditional Memory Balloon Device Conformance} > + > +A traditional memory balloon device MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{devicenormative:Device Types / Memory Balloon Device / Feature bits} > +\item \ref{devicenormative:Device Types / Memory Balloon Device / Device Operation} > +\item \ref{devicenormative:Device Types / Memory Balloon Device / Device Operation / Memory Statistics} > +\end{itemize} > + > \subsection{SCSI Host Device Conformance}\label{sec:Conformance / Device Conformance / SCSI Host Device Conformance} > > An SCSI host device MUST conform to the following normative statements: > @@ -292,7 +312,7 @@ Feature Bits / Legacy Interface: A Note on Feature Bits} > \item Section \ref{sec:Device Types / Block Device / Device Operation / Legacy Interface: Device Operation} > \item Section \ref{sec:Device Types / Console Device / Device configuration layout / Legacy Interface: Device configuration layout} > \item Section \ref{sec:Device Types / Console Device / Device Operation / Legacy Interface: Device Operation} > -\item Section \ref{drivernormative:Device Types / Memory Balloon Device / Device Operation} > +\item Section \ref{sec:Device Types / Memory Balloon Device / Feature bits / Legacy Interface: Feature bits} > \item Section \ref{sec:Device Types / Memory Balloon Device / Device Operation / Legacy Interface: Device Operation} > \item Section \ref{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics} > \item Section \ref{sec:Device Types / SCSI Host Device / Device configuration layout / Legacy Interface: Device configuration layout} > diff --git a/content.tex b/content.tex > index 11015a5..3f728e4 100644 > --- a/content.tex > +++ b/content.tex > @@ -1,4 +1,4 @@ > -\chapter{Basic Facilities of a Virtio Device}\label{sec:Basic Facilities of a Virtio Device} > +{Basic Facilities of a Virtio Device}\label{sec:Basic Facilities of a Virtio Device} > > A virtio device is discovered and identified by a bus-specific method > (see the bus specific sections: \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus}, > @@ -1046,7 +1046,7 @@ Transitional PCI Device ID & Virtio Device \\ > \hline > 0x1001 & block device \\ > \hline > -0x1002 & memory ballooning (legacy) \\ > +0x1002 & memory ballooning (traditional) \\ > \hline > 0x1003 & console \\ > \hline > @@ -2966,7 +2966,7 @@ Device ID & Virtio Device \\ > \hline > 4 & entropy source \\ > \hline > -5 & memory ballooning (legacy) \\ > +5 & memory ballooning (traditional) \\ > \hline > 6 & ioMemory \\ > \hline > @@ -4357,14 +4357,13 @@ how many random bytes were received. > The device MUST place one or more random bytes into the buffer, but it > MAY use less than the entire buffer length. > > -\section{Legacy Interface: Memory Balloon Device}\label{sec:Device Types / Memory Balloon Device} > +\section{Traditional Memory Balloon Device}\label{sec:Device Types / Memory Balloon Device} > > -This device is deprecated, and thus only exists as a legacy device > -illustrated here for reference. The device number 13 is reserved for > -a new memory balloon interface which is expected in a future version > -of the standard. > +This is the traditional balloon device. The device number 13 is > +reserved for a new memory balloon interface, with different > +semantics, which is expected in a future version of the standard. > > -The virtio memory balloon device is a primitive device for > +The traditional virtio memory balloon device is a primitive device for > managing guest memory: the device asks for a certain amount of > memory, and the driver supplies it (or withdraws it, if the device > has more than it asks for). This allows the guest to adapt to > @@ -4393,6 +4392,26 @@ guest memory statistics to the host. > memory statistics is present. > \end{description} > > +\drivernormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits} > +The driver SHOULD accept the VIRTIO_BALLOON_F_MUST_TELL_HOST > +feature if offered by the device. > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits} > +If the device offers the VIRTIO_BALLOON_F_MUST_TELL_HOST feature > +bit, and if the driver did not accept this feature bit, the > +device MAY signal failure by failing to set FEATURES_OK > +\field{device status} bit when the driver writes FEATURES_OK into > +\field{device status}. > +\subparagraph{Legacy Interface: Feature bits}\label{sec:Device > +Types / Memory Balloon Device / Feature bits / Legacy Interface: > +Feature bits} > +As the legacy interface does not have a way to gracefully report feature > +negotiation failure, when using the legacy interface, > +transitional devices MUST support guests which do not negotiate > +VIRTIO_BALLOON_F_MUST_TELL_HOST feature, and SHOULD > +allow guest to use memory before notifying host if > +VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated. > + > \subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout} > Both fields of this configuration > are always available. > @@ -4404,25 +4423,32 @@ struct virtio_balloon_config { > }; > \end{lstlisting} > > -Note that these fields are always little endian, despite convention > +\subparagraph{Legacy Interface: Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device > +configuration layout / Legacy Interface: Device configuration layout} > +When using the legacy interface, transitional devices and drivers > +MUST format the fields in struct virtio_balloon_config > +according to the little-endian format. > +\begin{note} > +This is unlike the usual convention that was used for other legacy devices > that legacy device fields are guest endian. > +\end{note} > > \subsection{Device Initialization}\label{sec:Device Types / Memory Balloon Device / Device Initialization} > > +The device initialization process is outlined below: > + > \begin{enumerate} > \item The inflate and deflate virtqueues are identified. > > \item If the VIRTIO_BALLOON_F_STATS_VQ feature bit is negotiated: > \begin{enumerate} > \item Identify the stats virtqueue. > - > - \item Add one empty buffer to the stats virtqueue and notify the > - device. > + \item Add one empty buffer to the stats virtqueue. > + \item DRIVER_OK is set: device operation begins. > + \item Notify the device about the stats virtqueue buffer. > \end{enumerate} > \end{enumerate} > > -Device operation begins immediately. > - > \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / Device Operation} > > The device is driven by the receipt of a > @@ -4451,16 +4477,16 @@ configuration change interrupt. > \item If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the > guest informs the device of pages before it uses them. > > - \item Otherwise, the guest MAY begin to re-use pages previously > + \item Otherwise, the guest is allowed to re-use pages previously > given to the balloon before the device has acknowledged their > withdrawal\footnote{In this case, deflation advice is merely a courtesy. > }. > \end{enumerate} > > -\item In either case, once the device has completed the inflation or > - deflation, the driver updates \field{actual} to reflect the new number of pages in the balloon\footnote{As updates to device-specific configuration space are not atomic, this field > -isn't particularly reliable, but can be used to diagnose buggy guests. > -}. > +\item In either case, the device acknowledges inflate and deflate > +requests by using the descriptor. > +\item Once the device has acknowledged the inflation or > + deflation, the driver updates \field{actual} to reflect the new number of pages in the balloon. > \end{enumerate} > > \drivernormative{\subsubsection}{Device Operation}{Device Types / Memory Balloon Device / Device Operation} > @@ -4474,12 +4500,38 @@ The driver MUST use the deflateq to inform the device of pages that it > wants to use from the balloon. > > If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the > -driver MUST wait until the device has used the deflateq descriptor > -before using the pages. > +driver MUST NOT use pages from the balloon until > +the device has acknowledged the deflate request. > + > +Otherwise, if the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is not > +negotiated, the driver MAY begin to re-use pages previously > +given to the balloon before the device has acknowledged the > +deflate request. > + > +In any case, the driver MUST NOT use pages from the balloon > +after adding the pages to the balloon, but before the device has > +acknowledged the inflate request. > + > +The driver MUST NOT request deflation of pages in > +the balloon before the device has acknowledged the inflate > +request. > > The driver MUST update \field{actual} after changing the number > of pages in the balloon. > > +\devicenormative{\subsubsection}{Device Operation}{Device Types / Memory Balloon Device / Device Operation} > + > +The device MAY modify the contents of a page in the balloon > +after detecting its physical number in an inflate request > +and before acknowledging the inflate request by using the inflateq > +descriptor. > + > +If the VIRTIO_BALLOON_F_MUST_TELL_HOST feature is negotiated, the > +device MAY modify the contents of a page in the balloon > +after detecting its physical number in an inflate request > +and before detecting its physical number in a deflate request > +and acknowledging the deflate request. > + > \paragraph{Legacy Interface: Device Operation}\label{sec:Device > Types / Memory Balloon Device / Device Operation / Legacy > Interface: Device Operation} > @@ -4488,7 +4540,27 @@ When using the legacy interface, the driver SHOULD ignore the \field{len} value > Historically, some devices put the total descriptor length there, > even though no data was actually written. > \end{note} > +When using the legacy interface, the driver MUST write out all > +4 bytes each time it updates the \field{actual} value in the > +configuration space, using a single atomic operation. > +\begin{note} > +Historically, devices used the \field{actual} value, even though > +the device-specific configuration space was never guaranteed > +to be atomic. > +\end{note} > +When using the legacy interface, the device MUST NOT use the > +\field{actual} value written by the driver in the configuration > +space, until the last, most-significant byte of the value has been > +written. > +\begin{note} > +Historically, drivers wrote the \field{actual} value > +by using multiple single-byte writes in order, from the > +least-significant to the most-significant value. Or maybe: most-significant byte of the value. > +\end{note} > > +\footnote{As updates to device-specific configuration space are not atomic, this field > +isn't particularly reliable, but can be used to diagnose buggy guests. > +} > \subsubsection{Memory Statistics}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics} > > The stats virtqueue is atypical because communication is driven > @@ -4513,10 +4585,11 @@ as follows: > subsequent request) and consumes the statistics. > \end{enumerate} > > + Within the buffer, statistics are an array of 6-byte entries. > Each statistic consists of a 16 bit > tag and a 64 bit value. All statistics are optional and the > driver chooses which ones to supply. To guarantee backwards > - compatibility, the driver SHOULD omit unsupported statistics. > + compatibility, the device ought to omit unsupported statistics. > > \begin{lstlisting} > struct virtio_balloon_stat { > @@ -4526,17 +4599,67 @@ struct virtio_balloon_stat { > #define VIRTIO_BALLOON_S_MINFLT 3 > #define VIRTIO_BALLOON_S_MEMFREE 4 > #define VIRTIO_BALLOON_S_MEMTOT 5 > - u16 tag; > - u64 val; > + le16 tag; > + le64 val; > } __attribute__((packed)); > \end{lstlisting} > > +\drivernormative{\paragraph}{Memory Statistics}{Device Types / Memory Balloon Device / Device Operation / Memory Statistics} > +Normative statements in this section apply if and only if the > +VIRTIO_BALLOON_F_STATS_VQ feature has been negotiated. > + > +The driver MUST make at most one buffer available to the device > +in the statsq, at all times. > + > +After initializing the device, the driver MUST make an output > +buffer available in the statsq. > + > +Upon detecting that device has used a buffer in the statsq, the > +driver MUST make an output buffer available in the statsq. > + > +Before making an output buffer available in the statsq, the > +driver MUST initialize it, including one struct > +virtio_balloon_stat entry for each statistic that it supports. > + > +Driver MUST use an output buffer size which is a multiple of 6 > +bytes for all buffers submitted to the statsq. > + > +Driver MAY supply struct virtio_balloon_stat entries in the > +output buffer submitted to the statsq in any order, without > +regard to \field{tag} values. > + > +Driver MAY supply a subset of all statistics in the output buffer > +submitted to the statsq. > + > +Driver MUST supply the same subset of statistics in all buffers > +submitted to the statsq. > + > +\devicenormative{\paragraph}{Memory Statistics}{Device Types / Memory Balloon Device / Device Operation / Memory Statistics} > +Normative statements in this section apply if and only if the > +VIRTIO_BALLOON_F_STATS_VQ feature has been negotiated. > + > +Within an output buffer submitted to the statsq, > +the device MUST ignore entries with \field{tag} values that it does not recognize. > + > +Within an output buffer submitted to the statsq, > +the device MUST accept struct virtio_balloon_stat entries in any > +order without regard to \field{tag} values. > + > \paragraph{Legacy Interface: Memory Statistics}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics / Legacy Interface: Memory Statistics} > + > When using the legacy interface, transitional devices and drivers > MUST format the fields in struct virtio_balloon_stat > according to the native endian of the guest rather than > (necessarily when not using the legacy interface) little-endian. > > +When using the legacy interface, > +the device MUST ignore all values in the first buffer in the > +statsq supplied by the driver after device initialization. > +\begin{note} > +Historically, drivers supplied an uninitialized buffer in the > +first buffer. > +\end{note} > + > \subsubsection{Memory Statistics Tags}\label{sec:Device Types / Memory Balloon Device / Device Operation / Memory Statistics Tags} > > \begin{description} > -- > MST
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]