Thread Links Date Links
Thread Prev Thread Next Thread Index Date Prev Date Next Date Index

RE: [RPRWG] Comment Resolution Process

Bob Love,
 
Thanks for informing us all regarding the process.  This should help
in understanding the ramifications of voting throughout the
balloting process.  In the spirit of election day, there are a
couple things which I would like to see clarified in a revised
draft of the Comment Resolution Protocol.
 
 
1.  Please include points you made in earlier emails regarding the
comment resolution and balloting process (see attached).  It's
important for all aspects of the balloting process to appear in
one place. 
 
2.  Comment resolutions are voted on comment by comment basis of the
one's which have commentor objections.  Please identify which type of comment
objections are taken to the WG, (E, T, TR).  If all 3 are taken to
the WG, please include % of votes for each type required to approve
a particular comment resolution?  Please discuss how specific resolutions are handled
if fail to receive required % approval.  See 1st highlighted text.
 
3.  You also seem to imply that once voting has completed on individual comments
a subsequent vote is taken on the document as a whole (see 2nd highlighted
text).  Please be more explicit as to whether WG voting discussed here is of
the individual comments or document as a whole.
 
4.  You mention that ballots sent to LMSC normally have a 95% approval rating
at the WG.  If all the voting at the WG level is geared toward 75% approval,
I'm not sure how 95% is ever achieved?  It is also important to identify
the minimum acceptable WG approval for LMSC ballot if such a criteria
exists?
 
 
There are also several issues with the comment resolution database (CRD) based on
the comment resolution process you outlined here and in earlier messages.  Let me
know whether you think the following changes in the CRD need to be made.
Maybe there is someone who is MSAccess guru who can speedily incorporate
the changes into the database.
 
1.  Each record in the CRD does not include a document issue number
from which the comment was made.  As we go to new document revisions
and section numbering changes there is no way to uniquely
determine which comments are associated with which document sections.
For example, in T&D each term is a unique section.  As a new term is added
section numbers of the remaining terms are affected.  It shall be impossible
to track original comments made against these terms.  Having a document
revision # in the record, at least allows us to bring up the old draft and identify the section
against which the comment was originally made.  An alternative might
be to maintain a separate CRD for each document revision; although this
introduces it's own set of problems.
 
 
2.  The CRD does not support a field "to be reviewed by working group".
The CRD needs to be enhanced to include this field and the associated
reports we will need to generate.
 
3.  The CRD has commentor "closed, unsatisfied, withdrawn" instead of :
"accepted, rejected, accommodated, withdrawn".  While the wording
is a little different, we are missing the "accommodated" category.
Should the database be modified to include accommodated, or use
as is?
 
4.  The CRD does not have "Revised Resolution Group proposed Resolution" field.
Is this something which needs to be added or do we add subsequent updates
to the "response" field with the proper annotations?
 
 
5.  The voter list in the database should include a field for objectionable
comments.  This should include a list of those comment #'s which a voter
objects to the resolution.  This list may comprise of either comments
directly submitted by the voter or comments submitted by others.
The voting record needs to make provision for voting specific sections
vs. the entire document in cases where we are balloting a section.
This allows us to track when we see votes in the database what the
votes pertain to. The current voter list and voting record for T&D section 1
of draft needs to be added to the CRD.  Do we need to make the above
changes in the CRD to the voting record?
 
6.  There is a problem with sorting by section/subsection, in that MSaccess
performs an alphanumeric sort vs. numeric sort.  In this case you'll see
sections 1.31 preceding 1.4.  It was a pretty big nuisance
for resolving comments in the text.  I tried to fix this in a temporary copy
of the database, but changing the field from alphanumeric to numeric
resulted in all the values being lost.
 
Regards,
 
Bob Castellano
 
Robert Castellano
Jedai Broadband Networks
rc@jedai.com
(732) 758-9900 x236
-----Original Message-----
From: owner-stds-802-17@majordomo.ieee.org [mailto:owner-stds-802-17@majordomo.ieee.org]On Behalf Of RDLove
Sent: Sunday, November 04, 2001 8:39 AM
To: 802.17
Cc: 802 SEC
Subject: [RPRWG] Comment Resolution Process

All:  John Hawkins requested that we describe the comment resolution process in much the same way as we write a standard, that is by describing the protocol clearly and unambiguously.  I took a first cut at addressing his request and circulated my draft to the SEC for their comments.  As a result of feedback I received, here is the Comment Resolution Process as I understand it, and as reviewed by the SEC.
 

Comment Resolution Protocol

 After all comments on a standard draft are submitted and compiled the Comment Resolution group gets first crack at addressing them.  Every voting member that wants to be a part of the comment resolution group may do so.  Non-voting participants may also be allowed to be a part of the group, based on the discretion of the chair.  Understand that being a part of the group may involve traveling to an additional meeting or two, and/or participating in teleconferences, in addition to time spent studying the comments.

 Comments are normally grouped for ease in reviewing them.

 The comment resolution group attempts to understand each comment and seriously address both the words and any underlying concerns that they believe are behind the comments.  It is still the duty of the commenter to recommend the replacement text, or specifically what must be done to resolve the stated issue.

 If possible, the comment resolution committee should send their proposed comment resolutions to the original commenter (whenever they do not simply accept the proposed change).  Sometimes schedule pressures preclude this possibility prior to the entire working group reviewing the proposed comments.  The working group gets the opportunity to review the proposed comment resolutions and any email postings that may support or challenge some of the proposed resolutions.  The entire working group gets to vote to approve the comment resolutions on a comment by comment basis (If there is no objection, then no vote is taken.  If there is no request to review a comment resolution, then it is assumed that there is no objection to it.)

 For those comments that are stated to be mandatory to change a vote from Disapprove to Approve, the commenter is asked to approve the working group's approved comment disposition.  (In some working groups, the commenter is asked to approve all comment resolutions.)

 Based on comment resolution, some votes may change from Disapprove to Approve.  If after the comment resolution process is complete, at least 75% of voting members that cast an Approve or Disapprove vote, now vote Approve, the ballot is said to have passed.

Note that at any point, a commenter may withdraw a comment.  In that case, it is as if that comment had never been submitted.  The withdrawal of a comment may cause a voter to change his/her vote from Disapprove to Approve.

 What Happens Now?

 If the draft does not garner at least 75% approval after comment resolution, the ballot has failed, and a new draft is prepared for ballot. 

 If the draft does achieve a 75% ballot resolution, then we begin the next steps:

 Assuming technical changes were made to the draft during the ballot resolution process, and/or, if there are still unresolved negative votes, a revised draft is prepared based on the approved changes to the text.  That draft now goes out for “recirculation ballot”, a period that could be as short as 10 days.  Along with the draft, all Technical Required Comments that have not been addressed to the commenter’s satisfaction are also circulated.  The working group approves a statement that explains why the ballot comment was not addressed to the satisfaction of the commenter, and includes that statement with the comments circulated along with the draft.  The commenter has the right to develop a rebuttal statement to the one approved by the working group, stating why the proposed resolution falls short of what is required.  In this case, the rebuttal is included in the recirculation.  The Working Group reply to the comment and any rebuttal remain with the document all the way through the RevCom approval submission or until the commenter withdraws his Disapprove vote for that issue.

During the recirculation, each person that voted on the originally balloted draft has a chance to change their vote based on changes made to the document, and based on the information obtained by reading the Required Technical Comments that were not addressed to the commenter’s satisfaction.  (Note: The recirculation process does not grant voters the right to now look closely, for the first time, at portions of the document that were not changed, but were just not reviewed well during the original ballot period.)  If someone that voted on the initial draft does not respond to the recirculation vote, then the voter’s Approve, Disapprove, or Abstain on the initial ballot remains unaltered.

 All comments made during a recirculation vote are handled the same way that comments were treated during the initial vote on the draft.  In addition, the voter list for recirculation remains unchanged during the entire recirculation process.

If, following the recirculation ballot, there are no new Negative votes, and no new comments that result in technical changes to the draft being made, then the Working Group ballot process is complete and the draft can proceed to LMSC Sponsor level ballot.

If there are new unresolved comments, or further technical changes made to the draft as a result of comment resolution, then another recirculation must take place.  Recirculation ballots continue until there are no new negative votes, and no technical changes to the draft as a result of comments made.  At this point the draft is ready for LMSC Sponsor level ballot. Normally ballots sent to LMSC Sponsor level have an approval rating at the working group level of at least 95%.                

                                  - - - - - - - - - - - - - - - -

 

 
 
Best regards,
 
Robert D. Love
Chair, Resilient Packet Ring Alliance
President, LAN Connect Consultants
7105 Leveret Circle     Raleigh, NC 27615
Phone: 919 848-6773       Mobile: 919 810-7816
email: rdlove@ieee.org          Fax: 208 978-1187

Bob, these are good questions.  Let me review the process in some detail including answers to your questions.
 
The data base that gets published and printed out for review by the working group needs to have the following fields:
 
Commenter
Comment Number
Editorial / Technical / Technical Required
Must be Reviewed by Working Group (This box can be checked by the commenter or by the Resolution Group - We didn't have this field in our first ballot.  It is there because the commenter may want to indicate an editorial change is important enough that it really needs broad review, or the resolution group may feel uncomfortable not having a particular proposed resolution get extra scrutiny, or the commenter will not be at the meeting but has expressed concern with the proposed resolution, or significant concerns were posted on the reflector and the resolution group wants to make sure those concerns are reviewed.)
Original Concern
Original Proposed Resolution
 
Comment Accepted / Rejected / Accommodated / Comment Withdrawn  (One of the four will be checked.  Accepted is only checked if specific proposed wording is accepted.  Accommodated means that it is believed the resolution may satisfy the concern of the commenter.  Sometimes after discussing the concern with the commenter, the commenter agrees to withdraw the comment.)
 
Resolution Group's Proposed Resolution
 
Working Groups approved Resolution
 
There will be a single character field  Commenter accepts Resolution.
 
Note that if through some process you come to a resolution the resolution group and commenter agree on, then the Resolution Group's Proposed Resolution is accepted by the commenter.
 
Once that data base report is circulated all WG members will have a chance to comment on the Resolution Group's proposed resolution.  That includes the original commenter (Ideally, it would be nice to get the commenter's input before publishing the data base.  In this case we have too little time to do that.)  Anyone with concerns over the proposed comment resolutions should post their concerns to the reflector.  Based on reviewing those concerns, the comment resolution group may decide to change the Resolution Group's Proposed Resolution prior to or at the meeting.
 
As a bonus to the WG, if you can have a copy of the proposed updated T&Ds with highlighted cross reference from each change to the commenter/comment number it is addressing(a single change may reference multiple comments, and a single comment may force multiple changes), that would allow people to see the implications of the proposed resolutions.  That document could be posted after the data base is posted, if it can't be ready at the same time.
 
= = = =
 
Given that part of the process, at the meeting you will need to have available an updated data base with an additional field "Revised Resolution Group's Proposed Resolution"  When reviewing comments with the group we quickly go through the data base asking people to state which proposed resolutions they have concerns with.  Only those comments, and ones with "must be reviewed by the WG" will be reviewed by the WG.
 
All comments with proposed resolutions that go unchallenged become WG approved resolutions.  The working group votes on approved wording when there is disagreement on approved resolution.  a 75% approval is needed for technical issues.
 
Mike, please review carefully and let me know if you have any concerns with this proposed process.
 
John, you are on copy because the process impacts the design of the data base.
 
Best regards,
 
Robert D. Love
Chair, Resilient Packet Ring Alliance
President, LAN Connect Consultants
7105 Leveret Circle     Raleigh, NC 27615
Phone: 919 848-6773       Mobile: 919 810-7816
email: rdlove@ieee.org          Fax: 208 978-1187
----- Original Message -----
Sent: Thursday, October 18, 2001 7:14 PM
Subject: Comment Resolution with Working Group

Bob Love,
 
Bob Sultan relayed to me your discussion regarding
RS and myself to finish putting our proposed resolutions
into the database and send out to the working group.  When
we send to the working group, how do we log the responses?
I see 4 types of responses coming back from the working group.
 
1.  The comment originator agrees with our proposed resolution.
2.  The comment originator doesn't agree with our proposed resolution, but
through some process we come to a mutual resolution.
3.  We are not able to come to agreement on a mutual resolution with
the comment originator.
4.  Some other member of the working group does not agree with our proposed
resolution that either the commentor originator already agrees with or
doesn't agree with.
 
It's a given that we will strive for consensus while trying to maintain the
technical and editorial integrity of the document.
 
My question is how do we document or log in the database the above
kinds of responses? I see the database supports a response status field
"closed, unsatisfied, withdrawn".  Is this where the commentor's
final acceptance/rejection of the resolution logged?  What about
another member's dissatisfaction of the resolution? Does that constitute
a new comment that is separately tracked?
 
At what point do the dispositions transition from proposal mode
to final mode?  At what point can the T&D section go through a revision,
and how is the incorporation of changes in the revision,
reflected in the comment database?  My assumption is that
the database records are updated to final mode, following
final ballot.  I'm not sure any special updating of the database
is required to incorporate the comments aside from the document
reflecting the proposed resolutions.
 
Let me know if my assumptions about the process are correct.
 
        thanks,
 
        bob
 
Robert Castellano
Jedai Broadband Networks
(732) 758-9900 x236
 


Bob Castellano brings up some good questions that everyone ought to know the answers to, and probably does if they have been involved in IEEE 802 for 10 or more years. 
 
For those of you that have not mis-spent your last 10 years, here are the answers:
 
All NO votes must be based on issues brought up during the balloting period.  If all issues that have caused a voter to issue a NO vote have been addressed to that voter's satisfaction, then the voter no longer has a valid NO vote.  However, the voter must state that the issues have been resolved before his no vote is converted. 
 
There is no requirement for the working group to address issues brought up after the ballot period closes.  Note, however, that in the process of resolving comments, additional errors in the standard are often uncovered by the ballot resolution group / editors.  In IEEE 802.5 we had a separate category for new issues that were uncovered during ballot resolution and the working group always worked to resolve these issues, although they were not associated with NO votes. 
 
If a commenter votes NO without providing specific changes for converting that NO vote, then the NO vote may be declared invalid and thrown out.
 
Balloters may not register NO votes based on issues uncovered after the ballot has closed.  However, it is acceptable to change your vote to a NO during a recirculation ballot based on information learned from reading the other comments, and / or based on changes made to the draft to resolve issues.  This capability is what drives the rules for recirculation ballots.
 
The working group can decide to address new issues not uncovered during the voting or ballot resolution process, but is under no obligation to do so.  They are, however, leaving the draft subject to being rejected by voters during the LMSC ballot.  Therefore, concerns with the standard are normally addressed fully prior to LMSC balloting.
 
Best regards,
 
Robert D. Love
Chair, Resilient Packet Ring Alliance
President, LAN Connect Consultants
7105 Leveret Circle     Raleigh, NC 27615
Phone: 919 848-6773       Mobile: 919 810-7816
email: rdlove@ieee.org          Fax: 208 978-1187
----- Original Message -----
Sent: Thursday, September 20, 2001 11:51 AM
Subject: RE: [RPRWG] IEEE 802.17 Balloting Instructions

John,
 
No problem.  I just wanted to make sure it wasn't a "Netscape problem", similar
to the ones I had before.   I think it would be really useful to writeup the
balloting process, so that people clearly understand the implications of
their vote.  Also, understanding the comment resolution process.  Understanding
scenarios like, if all the TR comments of a commentor are addressed are they obligated to
change a NO vote to a vote to approve.  How is a commentor's "NO vote"
"officially" handled if they do not provide any TR comments.  Does
it stand as a "NO", changed to a "Yes", or thrown out?  Are commentor's
allowed to continue to vote "NO", if their original comments have been
addressed, but they bring up a new reason which was previously in
the document that was balloted and not due to a change from the
previous version.
 
 
 
        thanks,
 
        bob
 
Robert Castellano
Jedai Broadband Networks
(732) 758-9900 x236
-----Original Message-----
From: John Hawkins [mailto:jhawkins@nortelnetworks.com]
Sent: Thursday, September 20, 2001 10:31 AM
To: 'rc'; RDLove
Subject: RE: [RPRWG] IEEE 802.17 Balloting Instructions

Hi Bob,
 
You don't see the file 'cause I never created it!  One of those things that fell through the proverbial crack. I know Bob sent out several emails with instructions and clarifications. Also, the ballot itself was built to be self-explanatory (wishful thinking I know).
 
Before we vote again, I'll compose a more comprehensive "help" doc for this purpose. I won't undertake that right now as there is no vote underway.
 
john
-----Original Message-----
From: Robert Castellano [mailto:rc@jedai.com]
Sent: Monday, September 17, 2001 6:53 PM
To: RDLove
Cc: Hawkins, John [WWP1:2268:EXCH]
Subject: RE: [RPRWG] IEEE 802.17 Balloting Instructions

Bob,
 
I receive an error when I try this link.  This may be one of the reasons
why people are not following the process as requested.
 
        thanks,
 
        bob c.
 
Robert Castellano
Jedai Broadband Networks
(732) 758-9900 x236
-----Original Message-----
From: owner-stds-802-17@majordomo.ieee.org [mailto:owner-stds-802-17@majordomo.ieee.org]On Behalf Of RDLove
Sent: Thursday, August 23, 2001 11:28 AM
To: 802.17
Subject: Fw: [RPRWG] IEEE 802.17 Balloting Instructions

All, on August 10th I posted the balloting instructions attached to the bottom of a long note.  Since then I have received two requests for instructions on posting the ballots.  Here are the concise balloting instructions for the Terms and Definitions draft, without the excess verbiage contained in the first note.
 
(John Hawkins, please post the full balloting instructions at http://www.ieee802.org/17/documents/drafts/Ballot_instr.txt, as indicated in the August 10th note) 
 
Best regards,
 
Robert D. Love
Chair, Resilient Packet Ring Alliance
President, LAN Connect Consultants
7105 Leveret Circle     Raleigh, NC 27615
Phone: 919 848-6773       Mobile: 919 810-7816
email: rdlove@ieee.org          Fax: 208 978-1187
 
 
- - - - - - - - - - - - Balloting Information - - - - - - - - - - - - - -

PLEASE RESPOND NO LATER THAN August 31, 2001
COMMENTS WILL NOT BE ACCEPTED AFTER September 8, 2001
RETURN ALL BALLOTS & COMMENTS BY E-MAIL TO:

E-MAIL: jhawkins@nortelnetworks.com and bob.sultan@fnc.fujitsu.com 

----- Please use 1 of the following 4 Subject lines on your email ballot --------

P802.17 TD1.0 RESPONSE = APPROVE WITH COMMENT
P802.17 TD1.0 RESPONSE = APPROVE WITH COMMENT
P802.17 TD1.0 RESPONSE = DISAPPROVE [comment(s) enclosed]
P802.17 TD1.0 RESPONSE = ABSTAIN 

----- Begining of Ballot Form.  Cut and paste text below this line only please --------

TA Document IEEE802.17-11Jul2001/:7, July 11, 2000
Working Group Ballot August, 2001
(Terms and Definitions for Resilient Packet Ring)


Your Name:___________________________


___    APPROVE WITHOUT COMMENT

___    APPROVE WITH COMMENTS

___    DO NOT APPROVE (Please attach specific comments for remedy)

___    ABSTAIN,   List Reason i.e. Lack of Expertise, Lack of Time:  ____________________________

------ END OF BALLOT SECTION -------

IF YOU ARE VOTING "APPROVE WITHOUT COMMENT" or ABSTAINING, YOUR WORK
IS DONE AT THIS POINT.

COMMENT INFORMATION:

COMMENTS ARE APPROPRIATE IN THE INSTANCE OF AN "APPROVE WITH COMMENT"
VOTE AND REQUIRED IN THE INSTANCE OF A "DO NOT APPROVE" VOTE.
"DO NOT APPROVE" VOTES MUST BE BACKED UP BY COMMENTS CLASSIFIED AS
"TECHNICAL REQUIRED" AND THESE MUST PROVIDE SUFFICIENT REMEDY AS
TO CHANGE THE VOTE TO AN "APPROVE" IF ADOPTED.  ALL COMMENTS SHOULD
INCLUDE CHANGES REQUIRED TO ELIMINATE THE STATED CONCERN.

PLEASE USE THE FORM BELOW FOR ALL COMMENTS. MULTIPLE COMMENTS CAN BE
SUBMITTED IN ONE ASCII TEXT FILE OR E-MAIL. SIMPLY ADD YOUR PERSONAL DATA
AND THEN COPY THE LINES FOLLOWING        "------ COMMENT SECTION -------"
AND REPEAT AS OFTEN AS REQUIRED WITHIN THE FILE OR EMAIL. PLEASE USE THE
FORMAT BELOW, MAKING SURE THAT THE COMMENTS ARE IN ASCII FORM AND THAT
EACH COMMENT INCLUDES: 
CommenterName:
CommenterEmail:
CommenterPhone:
CommenterCellPhone:
CommenterCompany:

Acceptable comment types:
E  = Editorial
T  = Technical
TR = Technical Required

------ COMMENT SECTION -------
Comments on P802.17/TD1.0

CommenterName:
CommenterEmail:
CommenterPhone:
CommenterCellPhone:
CommenterCompany:
Clause: 
Subclause:
Page:
Line:
CommentType (E, T or TR):
Comment #: 
Comment:


CommentEnd:
SuggestedRemedy:


RemedyEnd:

- - - - - - - - - - END OF BALLOT FORM - - - - - - - - - - - -

If you should have any questions, problems or comments please contact:

Mike Takefman
Chair, IEEE P802.17 Working Group
Office: 613-271-3399
takf@cisco.com

Bob Love
Vice-Chair, IEEE 802.17 Working Group
Office: 919-848-6773  Fax: 208-978-1187
rdlove@ieee.org

Bob Sultan
Editor, Terms and Definitions Section for IEEE 802.17 Working Group
Office:  845 731-211
bob.sultan@fnc.fujitsu.com

John Hawkins
Co-Webmaster, IEEE 802.17 Working Group
Office: 770 705-708-7375
jhawkins@nortelnetworks.com