stds-802-16: evaluation system and scoresheets for PHY and MAC proposals

["Roger B. Marks" <marks@boulder.nist.gov>]

In accordance with our approved "Development Plan for 802.16.1 Air
Interface Standard"
<http://grouper.ieee.org/groups/802/16/docs/99/80216-99_05.pdf>,

the PHY and MAC proposals presented next week will be evaluated and
scored by the 802.16 Voting Members. The submitters of proposals
averaging 6 or better (out of 10) in _any_ evaluation category will
be invited to submit a more detailed proposal for Session #5 in
January. The evaluation categories were defined in the Call for
Contributions. For the PHY, you can look in either of these places:

http://grouper.ieee.org/groups/802/16/phy/docs/80216p-99_01.pdf
http://grouper.ieee.org/groups/802/16/phy/docs/80216p-99_01.html

For the MAC, see:

http://grouper.ieee.org/groups/802/16/mac/docs/80216m-99_01.pdf
http://grouper.ieee.org/groups/802/16/mac/docs/80216m-99_01.html


Regarding the scoring systems, I have already posted:

Proposal Evaluation Form and Scoring Process
<http://grouper.ieee.org/groups/802/16/docs/99/80216-99_07.pdf>

According to this plan, the evaluators' scores will not be
renormalized in any evaluation category unless the average deviation
from 5 (out of 10) is greater than 2.5 in that category. There has
been some discussion regarding the merits of this idea. I am open to
improvements in the future. However, the Chair is responsible for
procedural matters, and I consider this matter procedural. I am
declaring that we will try out this deviation cap for Session #4,
although not necessarily beyond. The score submission forms include a
section for comments regarding the process.

I have created customized scoresheets that allow easy data entry and
export to CSV (comma separated variable) text files. These
scoresheets, in Microsoft Excel, are available now in both PHY and
MAC forms:

http://grouper.ieee.org/groups/802/16/docs/802_16_PHY_eval.xls
http://grouper.ieee.org/groups/802/16/docs/802_16_MAC_eval.xls

You may begin using these spreadsheets as you read the proposals. You
can fine tune them during the meetings. The submission deadline is
12:30 pm on Thursday, November 11, just after the Thursday morning
PHY and MAC discussion sessions close. Please let me know right away
if you find any typos or functional problems with the scoresheets.

To submit your scores, you will save two CSV files (one each for PHY
and MAC) and submit them. The files can be submitted on floppy, via
flash ROM, or via a 10Base-T network that Brian Petry has graciously
agreed to provide. Brian will be bringing a 48 port hub with 48
10Base-T cables. If you use the LAN, you will have to set your IP
Network Address to one of the designated LAN numbers (10.0.0.1-254).
We'll work out the details and create some instructions.

If you don't have a laptop or can't write to any of these formats,
you will have to borrow someone's equipment.

Brian has also written a PERL script to automatically read the
spreadsheets and calculate the averages. The results will be written
out to a file so that all voting members can confirm the accuracy of
their scores. We will need to get this done quickly so that we can
act on the results in the Closing Plenary.

If you think this is a lot of trouble, I agree with you. However, we
decided to do it in order to get feedback on the specific pros and
cons of the proposals. The intent is that this leads to better
proposals. We are trying to make the process as easy as possible for
the evaluators, so please hang in there and give it your best shot.

Thanks,

Roger


Dr. Roger B. Marks  <mailto:marks@nist.gov>
Chair, IEEE 802.16 Working Group on Broadband Wireless Access
National Wireless Electronic Systems Testbed (N-WEST) <http://nwest.nist.gov>
National Institute of Standards and Technology/Boulder, CO
phone: 1-303-497-3037  fax: 1-303-497-7828