Re: Link Status thoughts
Pat,
Thanks for all your comments. I looks like we're nitting picks so I'll
only address the very specific picks and provide far more detail and
real pics in my presentation (which I'm working on).
> In my opinion, each of three signals: the idle stream, the remote
> fault stream, and, if adopted, the local fault stream should be
> be sufficient for link initialization. At this point, all fault
> signaling schemes on the table have that property. Lock can even be
> obtained while receiving a mixed stream of idle and packets though it
> may take a bit longer.
I wholeheartedly agree with all of your statements above. These
statements form the foundation of 10GE link initialization, link status
reporting and the protocol which maintains robust link-level operation
in the presence or absence of packets. Now on to the nits. (P.S. The
reader is advised to consider the statements above while considering any
discussion which follows.)
> Remote Fault indications should not cause any trouble and reset
> timers shouldn't be necessary to ensure that.
Agreed on the first point, not on the second. I don't advocate requiring
reset timers. However, a robust implementation will likely include them
because they filter out a lot of "protocol noise". In general, a DTE
should not transmit any error reports until the DTE is reset, this
certainly includes acquiring sync is a signal is present. The simple
rule being followed here is "listen before you speak".
> 3) Device B powers up. By the time it has finished reset, it may or may
> not have achieved sync. Therefore it may start sending Remote Fault
> or Idle. We don't know and it doesn't matter.
The above is an example of "speak before you listen". Remote Fault
consists of both the detection of a Local Fault condition or Local Fault
report at the Local Device, and the generation and transmission of of a
Remote Fault message to its Link Partner. Therefore, I question the
benefit and even the sequence of events during reset which results in a
DTE sending Remote Fault before it even attempts to achieve sync at its
receiver. That said, I agree with your conclusion: "We don't know and it
doesn't matter." The latter means that any DTE may generate and transmit
a Remote Fault message whenever the required conditions exist.
> The bring up has to also work when both devices are up but not connected
> and the connection. It has to work when a nasty noise hit causes one or
> both sides to lose lock. Therefore, it is important that it not rely
> on one of them being reset. It has to come up when both sides have detected
> a fault and are sending remote fault.
Absolutely... and it does. Please note that the seven step
initialization between two DTE's in my last note described only the very
simple case of one DTE powering up before the other. I should have been
more specific and said "one day" before the other.
The point I made immediately after the seven step initialization is
that: "Break Link is not needed during Link Initialization and only
serves to complicate link initialization and link status reporting." It
does so by adding unneeded handshakes. Can I interpret your comments as
agreement with my observations with respect to Break Link?
> I have not been following Fibre Channel so I don't know the details of your
> proposal that was made there. There appear to me to be two choices to insert
> a fault indication with random /A/ spacing. Either replace the /A/ with the
> fault indication or put the fault indication in after the /A/. I slightly
> prefer the latter because then the sync process only has to align on /A/
> and it can align in the presence of the fault indication.
I'll explain Sequence and Signal in more detail here. BTW, Sequence and
Signal are both shown in the Clause 48 state machines (with an editorial
note indicated that this protocol has not been approved by the Task
Force).
Signal consists of the ordered-set /K/D/D/D/. This is the same format as
Start over the XGMII, 8B/10B PCS (including XAUI/XGXS) and 64B/66B PCS.
The choice of /K/ distinguishes Signal from Start and any other
ordered-sets with the same format. Signal may be inserted into the Idle
stream and it's insertion rate would be controlled to insure that the
robustness of the Idle protocol is not compromised. LSS uses Signal as
it's transport and proposes a periodic reporting of Remote Fault at 125
+/- 4 usec. Multiple occurrences of a particular Signal may be required
to validate the receipt of the message contained therein (e.g. Remote
Fault). The Signal message may be singular or plural. An example of a
plural Signal message would be an LSS message indicating both Remote
Fault and Local Fault. Multiple Signal messages may be concurrently
active.
Sequence utilizes the same ordered-set Format as Signal and Start,
/K/D/D/D/. The choice of /K/ distinguishes Sequence from Signal, Start
and any other ordered-sets with the same format. Only one Sequence may
be active at any time. Sequence replaces Idle at the XGMII and 64B/66B
PCS and alternates with Idle at the 8B/10B PCS (including XAUI/XGXS).
For the 8B/10B PCS, a continuous Sequence alternates with the random
||K||R|| Idle sequence between ||A||. Sequence is generally employed
when packet transmission is prohibited. I don't currently envision any
scenarios where Sequence would be used when packet transmission is
allowed for either 10GE or 10GFC. Multiple occurrences of the Sequence
ordered-set would required to recognize the Sequence message contained
therein (e.g. Remote Fault), I suggest 3 or 8. The Sequence message may
be singular or plural. An example of a plural Signal message would be a
Sequence message indicating both Remote Fault and Local Fault. The
following is an example of an 8B/10B PCS pure Idle sequence contrasted
to 8B/10B PCS Sequence protocol:
8B/10B PCS pure Idle sequence:
Lane 0 KKRARKRKKKRRKRRRKRRRKKRKRRAKRKRKRRKKKRRKRKRKKRRKRARR
Lane 1 KKRARKRKKKRRKRRRKRRRKKRKRRAKRKRKRRKKKRRKRKRKKRRKRARR
Lane 2 KKRARKRKKKRRKRRRKRRRKKRKRRAKRKRKRRKKKRRKRKRKKRRKRARR
Lane 3 KKRARKRKKKRRKRRRKRRRKKRKRRAKRKRKRRKKKRRKRKRKKRRKRARR
8B/10B PCS Sequence protocol
Lane 0 KKRAKKKKKKKKKKKKKKKKKKKKKKAKRKRKRRKKKRRKRKRKKRRKRAKK
Lane 1 KKRDDDDDDDDDDDDDDDDDDDDDDDAKRKRKRRKKKRRKRKRKKRRKRADD
Lane 2 KKRDDDDDDDDDDDDDDDDDDDDDDDAKRKRKRRKKKRRKRKRKKRRKRADD
Lane 3 KKRDDDDDDDDDDDDDDDDDDDDDDDAKRKRKRRKKKRRKRKRKKRRKRADD
> I do not see any point to being able to send RF and LF at the same time.
> To detect RF you need to have obtained sync to the incoming signal.
> Therefore, if you detect RF in band then an LF condition does not
> exist. If an LF condition exists you cant detect RF. They are mutually
> exclusive.
You are correct and I misspoke. Both conditions can simultaneously
exist, but can't be detected. In essence, the complex case of both
conditions existing simultaneously would be recognized as a Local Fault
condition. Therefore, Local Fault take s precedence over Remote Fault.
It would seem that MDIO can retain any Remote Fault information
conducive to the fault isolation process.
> I can see one purpose to a Break Link signal. In a perfect world, it
> wouldn't be needed. In an imperfect world, there may be a case where
> the something in the remote node has locked up and the remote node
> isn't detecting that it has a problem. For instance a jabber problem.
> I could see allowing a Break Link signal initiated by management to
> cause a reset to the device at the other end of the link using the
> lowest possible level of communication. I'm not convinced it is needed,
> but if people want it, I'd be willing to allow a mechanism for it.
I agree that the obscure case of a Local Device's receiver being in sync
but recognizing "jabber" may exist. In this case, a management initiated
Link Partner reset function may come in useful. What is clear is that
this function NEED NOT be tied to any DTE power-on or reset management
function as is the case of the current Break Link protocol. I suggest
that that management can invoke the transmission of a Remote Fault
message to accomplish this function. I can't fathom that the Link
Partner would respond any differently to a Remote Fault message than a
Break Link message, do you? If not, then no additional message is
required, just add a management control bit.
> Toward the end of your memo, you assert that every link element must
> be able to recognize a local fault condition and forward it and seem
> to imply that it must be done in-band. I don't think it is reasonable
> for 10GBASE-R PMDs and PMAs to do this inband. They would need almost
> all of the PCS layer to do it. Even for 10GBASE-X PMDs and PMAs it
> seems an excessively burdensome requirement. We should specify a pin
> at the XSBI for forwarding this information.
There's a HUUUUGGGGE difference between "must" and "may"/"should". A
careful re-read of that section of my note, quoted immediately below,
shows that I carefully chose to use the word "may" for recognition of
Local Fault, not "must" and "should" for recognition an forwarding as an
assertion that those devices which do so provide reliability,
availability and serviceability benefits over those which do not. I then
carefully go on to say that 10GE should specify the protocol by which
Local Fault and Remote Fault conditions are recognized and corresponding
message transport protocol defined. My point is that the protocol for
Fault condition recognition and in-band Fault message transport should
be defined in P802.3ae. It's use would NOT be required.
I don't see a problem with a pin or pins at the XSBI to transport Fault
condition(s). However, I view these Fault conditions as bypassing the
PMA as they are not applicable at that level. As an example, please see
Figure 48-2.
"On to Local Fault: Local Fault, as you suggest, is the condition and
signal associated with the Loss_of_Signal or Loss-of-Sync, etc.
conditions. Local Fault may be recognized and signaled up stream by any
link element including intermediate link elements such as an XGXS, PMD
retimer, WIS, etc. *** Note that this is a key decision point *** 10GE
architecture already specifies intermediate link elements. A 10GE link
may employ multiple such link elements. Each of these link elements
should be capable of both recognizing and forwarding Local Fault
conditions. If not, these elements become part of the problem rather
than the solution. The next question is whether the Local Fault signal
is in-band or out-of-band. Since pins=BAD, this seems like an
implementation issue, and Local Fault needs to be able to go over the
medium, it seems that Local Fault must at least be specified as an
in-band signal. The signaling requirements for Local Fault appear to be
identical to those of Remote Fault. Consequently, I propose that the
Local Fault signaling protocol be either LSS or an alternating
Idle/Sequence."
> I generally support Shimon's proposal, but I would like to eliminate
> the handshaking aspect. There just hasn't been time to get full concensus
> on that.
I'm strongly opposed to Shimon's current proposal. It's way too complex
in the problem set it chooses to address and in the solution to that
problem set. In general, Idle alone works just fine for 10GE link
initialization. The simple addition of Fault recognition and one-way
signaling is all that is required for a complete link initialization and
link status reporting protocol.
> Also, there has been a lot of focus on how to encode the signals but really
> there are a lot of other issues that we really have just been coming
> to understand like where does fault sensing live?
I believe I covered these issues in a previous note. "Fault sensing"
lives in the form of signal_detect circuitry at a serial link receiver.
It is generally not applicable to the signals of a parallel link/bus
including all those specified as P802.3ae interfaces.
> These discussions especially over the past week have been very helpful
> and I'm happy to work with all parties on coming to a concensus
> proposal.
I completely agree. I'm sorry that other pressing matters have kept me
to busy to allocate sufficient time to work these issue with you and
Shimon and Howard prior to the Tampa meeting. Is there a time we could
meet before presentation time? How about sometime between the end of the
802.3 session and Monday dinner to start with?
--
Best Regards,
Rich
-------------------------------------------------------
Richard Taborek Sr. Phone: 408-845-6102
Chief Technology Officer Cell: 408-832-3957
nSerial Corporation Fax: 408-845-6114
2500-5 Augustine Dr. mailto:rtaborek@xxxxxxxxxxx
Santa Clara, CA 95054 http://www.nSerial.com