At the October, 1998 meeting of the ACM Special Interest Group (SIG) Governing Board (SGB) Executive Committee (SGBEC--whew!) meeting, I, in my role as the SGBEC member responsible for conferences, recommended that the SGB look into conference management software that could be used in the 80-odd conferences the SIGs put on yearly.

As a first step, I sent out queries requesting information on the software systems that are currently in use. I was surprised at the number and capabilities of the systems that surfaced in this survey.

I thank the developers and others who provided the information contained herein. Corrections and comments are welcome; please send them to me.

Rick Snodgrass, rts@cs.arizona
January, 1999

Columns:

• 1. Submission of papers
• 2. Assignment to reviewers
• 3. Mailings to PC members
• 4. Submissions of reviews
• 5. Reviewer comment threads
• 6. Distributed PC meeting
• 7. Letters to authors
• 8. Conference registration
• 9. Handles associated workshops
• 10. Handles different tracks of submissions
• 11. Status: prototype: 1 to production quality: 3
• 12. Used in # of conferences
• Lang: Development language

• Y: Supports that feature
• -: does not support that feature
• ?: feature unclear, or partial support
• S: to be supported soon
• P: partial

Commercial software:

SIGCHI uses Web-based reviewing-management application that was written by
Lotus Research. They are now licensing it out through their Lotus Business
Partners office.

ACM Computing Reviews has first-class software for managing its review process.

• Agents '98
• Agents '99
• AAAI

Basically, there are three components:

1) PC registration of contact information and areas of expertise
2) Author submission of electronic title page and abstract
3) PC review of abstracts and bidding on papers to review

The data is then downloaded to a separate program, where an algorithm is run on it, which makes the reviewer assignments. This final piece of the software is not in our possession, although we currently have someone working on producing a version for us.

Our current software is copyrighted by Ramesh Patil, and is being revamped and reconfigured by Vibhu Mittal for AAAI. It is available for use through our conference services. Interested parties would have to contract with us to manage their paper processing in order to use this software. (Please note that AAAI's conference services are limited to conferences with some affiliation to the AI community.)

For parties not interested in contracting with AAAI, they can contact Ramesh Patil directly, who may be willing to make a copy available to them. However, the program is not of commercial quality, and does require a fair amount of tweaking and management as the process unfolds.
 

[Top of Page]

Overview: The application was built in conjunction with the ACM bibliographic data entry system that produces Guide. Both the Biblio and CR system run off of Sybase and is mostly application  driven. The Arborext software is used to produce the printed versions of Guide and CR and not used in the tracking process.

Portability: The data should be easily portable to Oracle or some other DB. The tracking application will probably be a difficult port since most of the functionality wasn't built into the DB.

[Top of Page ]

Developer(s): Paal Halvorsen, Ketil Lund, Thomas Preuss, Vera Goebel, Thomas Plagemann, Hartmut Kónig
Contact name and email address: Thomas Plagemann <plageman@unik.no>
URL: http://confman.unik.no (demo-version)
Source language: MSQL-lite
Platform(s): UNIX, linux
Used (conference name(s) and dates):

• DAIS'97 (IFIP WG 6.1 International Working  Conference on Distributed Applications and Interoperable Systems, Cottbus, Germany, 30.9.-2.10 1997)
• IDMS'98 (5th International Workshop on Interactive Distributed Multimedia Systems and Telecommunication Services , Oslo, Norway, 8.-11.9 1998)
• WOSP'98 (WORKSHOP ON SOFTWARE AND PERFORMANCE, Santa Fe, New Mexico, 12.-16.10 1998)
• MoMuC'98 (5th International Workshop on Mobile Multimedia Communications, Berlin, Germany, 12.-14.10 1998)
• Estelle'98 (International Workshop on the Formal Description  Technique Estelle, Evry, France, 2.11 1998)
• EDOC'98 (2nd International Enterprise Distributed Object Computing  Workshop San Diego, CA, 3.-5.11 1998)
• KiVS'99 (Kommunikation in Verteilten Systemen, Darmstadt, Germany, 2.-5.3 1999)
• NGHSN'99
• ACM SIGCOMM'99
• EDOC'99
• IDMS'99

Status(prototype, production quality): Version 1.0 is released and working, but some code might be optimized!?
Support provided: YES (limited):
1. we try to fix all reported problems (matter of available time)
2. an email list (confman-users@unik.no) for ConfMan users is created where users can subscribe and questions will be answered.

However, the installing the system should be very easy: No programming  is required; Everything is configurable from www; no access needed to the scripts; installation done by unpacking the package, changing two path-names in a configuration script and running this script.

There is a Technical Report that gives a good overview of design, implementation and evaluation of ConfMan. It can be found on the ConfMan site: http://confman.unik.no

Version 1.2 has been released. Since October 1998, there has been a lot of improvements due to comments from users. Please see http://confman.unik.no/~confman/whats_new.html for an update.

Appropriate for:
 

	Submission of papers:	Yes
	Assignment to reviewers:	Yes
	Mailings to PC members:	Yes
	Submissions of reviews:	Yes
	Reviewer comment threads:	Yes
	Distributed PC meeting:	Yes
	Letters to authors:	Yes
	Conference registration:	Yes
	Handles associated workshops:	Partial yes. There is no problem handling concurrent workshops by using one DBS server and multiple instances of the scripts. There should be no problem running parallel ConfMan-systems, i.e., only another name of the database is required. The workshops usually have own www-pages as well.
	Handles different tracks of submissions:	There is partial support for this feature now (only submission of papers by using different papertypes). You may define as many papertypes as you like. The list of papers sent to PC members include the papertype.

OBSession

Contact: Thomas Plagemann <plageman@unik.no>
Reporter: Murray Woodside <cmw@sce.carleton.ca>

[Top of Page]

• ECOOP'97, ECOOP'98, ECOOP'99 (http://ecoop.cs.utwente.nl, still running)
• ESEC/FSE 99 (http://www.iam.unibe.ch/~esec99/, still running)

Other features not mentioned:
       CyberChair generates several overviews of reviews:
           - For each reviewer:
              A list of all his/her reviews
              A list of all reviews for all papers (s)he reviewed
           - For the Chair:
              For each paper, a list of all reviews
              For each reviewer, a list of all his/her reviews

CyberChair generates for each paper that has at least one 'Champions'  (see the paper above), a list of all   champions (reviewers who indicated they would like to accept that paper very much).A status is shown that indicates how many reviews a reviewer still has to submit. The moment a reviewer is done, a big pie is automatically ordered online and sent to the reviewer. Reviewers who do not submit all reviews do not get a pie. However, a pie is ordered anyway, and sent to the maintainer of CyberChair.

• Infocom'99

[Top of Page]

• ICSE'99, May 1999

*Abstracts submitted electronically through Web; Papers submitted physically.

[Top of Page]

• SIGCSE '99, March 1999 (190 papers, 3 reviews each)

The last 3 items were NOT my department but could be accommodated.

[Top of Page]

PCA is great and could be used as an example of how to assign papers to reviewers, etc.

[Top of Page]

• SIGMOD (June 98),
• VLDB (Aug 98),
• POPL (98 I believe)

[Top of Page]

• WETICE97 Workshop on Collaborative Agents in Distributed Web Applications, June 18-20, 1997
• Minitrack on Coordination languages, models, systems in the Software Technology Track of the Hawaii International Conference on System Sciences (HICSS-31), January 6-9, 1998
• World Wide Web Journal Special issue on Distributed World Wide Web Processing:  Applications and Techniques of Web Agents, 1998
• WETICE98 Workshop on Coordination Architectures for Distributed Web Applications,  June 17-19, 1998
• Special issue of the Autonomous Agents and Multi-Agent Systems journal on Coordination Mechanisms and Patterns for Web Agents, 1998

Appropriate for :
 

	Submission of papers:	No
	Assignment to reviewers:	Yes
	Mailings to PC members:	No*
	Submissions of reviews:	Yes
	Reviewer comment threads:	?
	Distributed PC meeting:	No
	Letters to authors:	Yes
	Conference registration:	No
	Handles associated workshops:	No
	Handles different tracks of submissions:	Not sure*

Not sure*: if there are multiple disjoint tracks they would have to handles as difference "conferences" (like in HICSS). There is no support of putting/moving papers into slots of tracks.

No* means "to be implemented in next version".

[Top of Page]

•  SAC'99 WWW Track

Sort-of*: it collates review information and presents it to  the coordinator and other reviewers.

Yes and No*: reviews/verdicts/recommendations are collated for on-line presentation to authors.

Submission of papers:  Gathers author data and abstract. Allows papers to be FTPed to server.           Handles existence and abstracts for paper submissions

Assignment to reviewers:  it allows reviewers to see abstracts before agreeing to review papers.

Mailings to PC members:   It does, though, provide reviewers with access to  the list of all submissions
 the list of papers they are to review (+ their abstracts)  the bodies of those papers (as submitted on-line). It can also mail reminders to all reviewers, and  all reviewers with reviews outstanding.

Submissions of reviews: form for quantitative scores and free text  (confidential and for author)

Reviewer comment threads:  reviewers can see each other's reviews but only after the review submission deadline has closed.

Conference registration:   exports lists of authors and paper IDs, and of reviewers.

The software package used for the SAC'99 WWW track is a collection of Perl programs which are accessed through CGI.  It is configured by means of a small set of data files, and communicates by generating web pages and email messages.

This software performs the following tasks:

• accepted submission details
• allows papers (potentially in multiple files) to be uploaded by ftp and moving them out of harms way
• makes papers available to reviewers
• accepts reviews through a web form
• collates reviews and presented them to the track chair
• makes reviews and verdicts available to authors (and reviewers).

It is potentially available to anyone who is interested.

[Top of Page ]

• STOC
• FOCS
• SODA 3 years worth

[Top of Page]

[Top of Page]

• ACM TODAES Journal (Transaction on Design Automation and Electronic Systems).

We have been using the web for paper review in ACM TODAES Journal (Transactions on Design Automation of Electronic Systems) since September 1996. The system provides administrative tools online via the web for maintaining the papers for review (e.g., user accounts, restricting user access to papers being reviewed, automatic generation of HTML pages, etc.).

A modified version of this software is currently being used for submitting paper for review as well as reviewing papers via the web for the ACM/IEEE Design Automation Conference.

Currently there are plans being considered for a complete electronic paper submission/review for some/all conference sponsored by ACM.  As a result a version of the software currently being considered for this purpose. The software is UNIX based system and there is a plan to port the software to NT platform.

The software redundantly stores information both in a database format and a plain text file format to provide recovery in an event of system crash.

User access is restricted via Netscape's nsconfig protocol.

For the ACM TODAES Journal, it is combination of a ftp and web based system. This system have been evolved into a complete web based system for the ACM/IEEE Design Automation Conference.

[Top of Page]

• SIGGRAPH 99, November 1998 - May 1999

SIGGRAPH has been experimenting for a couple of years with different forms of electronic submission, each custom designed for use on the organization's Sun web server. New this year, the SIGGRAPH 99 conference is requiring electronic submission for all programs. The fully operational submissions system created by Q LTD, SIGGRAPH's web and design contractor, can be examined at http://www.siggraph.org/s99 by following the "Contributors" and then specific conference program links.

Q LTD is also in the process of creating an Online Review system for use by jury and review teams to evaluate program submissions.

[Top of Page]

[Top of Page]

Used (conference name(s) and dates):
 

• ACM Symposium on Parallel & Distributed Tools (SPDT 98), August 3-4, 1998

http://www.cs.uoregon.edu/research/SPDT98/,
Number submitted papers: 42
Number of reviews/edited reviews: 230
• 19th IEEE Real-Time Systems Symposium (RTSS 98),  December 1-4, 1998, http://www.cs.umd.edu/~rich/rtss98,    Number submitted papers: 182   Number of reviews/edited reviews: 862
• IEEE Workshop on High-Level Programming Models & Environments (HIPS '99) April 12, 1999

http://www.informatik.hu-berlin.de/~mueller/hips99
PC activity in progress.
• International Conference on Reliable Software Technologies, Ada-Europe'99,  June 7-11, 1999

http://www.ada-europe.org/conference99.html
PC activity in progress.
• International Conference on Computer-Aided,   Verification (CAV '99),    July 7-10, 1999

http://gaspard.imag.fr/CAV99
Submissions in progress.
• Euromicro '99,  June 11, 1999

http://europub.ict.tuwien.ac.at/rts.htm
Just getting started.

Status (prototype, production quality): Production quality.

Support provided:

• Free for unsupported use, with regular updates provided at no cost.  Most Computer Science or EE researchers could use this option, since installation/maintenance is basically brain-dead.
• Consulting/customizing services offered for larger events, on per-event contract basis.

(0) Multiple views are provided into the review database.  At some point in the process, the PC chair can set a toggle in the manager package. At that point, all reviewers of a paper see all other reviews on that paper - and then the debates can be channeled accordingly.

(1) Upload/auto-formatting of registration data, suitable for entry into MS Access, Excel, etc. For liability reasons, we do NOT provide a secure server with the package.  If secure server is required - i.e., certified for credit-card usage - then this service can be arranged with the developers on per-event basis.

(2) This is done transparently, via running multiple servers on different TCP ports.  The conference chairs don't need to know the mechanics of all this, nor do submittors/reviewers.  It's done automatically, by the START package.

(3) Currently this is done with multiple servers (again transparently).  In the next upgrade we will use one TCP port for all transactions.  To the conference manager, the semantics will be the same - but the new architecture will be easier for us to maintain.

NOTE: There are many features included in START that aren't covered in the questionnaire above. Please see the START homepage for details.

One item should be noted: We developed the full tool while I served as PC chair for RTSS98.  I also as de facto publicity chair for that conference. I designed START to amplify conference publicity wherever possible, and the results were better than expected.

For example, the keyword generator was built to export tags for search-engines.  It seems to have worked pretty nicely.  We also include a tool which produces an HTML list of accepted papers, and exports it on the conference web sit as soon as all decisions are made.  This tool also generates a directory with formated abstracts for each paper, linked to the list.  As soon as your publicity announcements go out, anybody can check out the conference abstracts.  Also, search engine robots start including one record for each abstract, all of which link back to your conference home page. (See http://www.cs.umd.edu/~rich/rtss98).  There are many similar widgets built in for this kind of thing.

These strategies help publicize your conference as much as possible, and target it to the audiences you want. The next version of START (V1.2) will include a manual on other techniques for E-publicity.

Using these techniques, RTSS 98 received twice as many papers as usual, and spend nothing on advertising the conference.  As of today, more people are PRE-registered for RTSS98 than attended RTSS97 - and the conference is still a month away.  That is to say: These methods work, and anyone who is building this sort of tool would be wise to consider the WHOLE conference management process. Submissions and reviews are only part of it.

[Top of Page]

*reviews for a given paper are grouped together on  a common web page

[Top of Page ]

• CASCON '97 November 1997 - IBM Research Conference, Toronto, Canada
• CASCON '98 December 1998
• WWW8 - W3C Conference, May 1999, Toronto, Canada
• 3DIM '99 - to be held in Ottawa, October 1999
• CASCON '99, November 8-11, 1999
• Will be used to run WWW9 - to be held in Amsterdam May 2000
• In discussion with Dr. Hausi Müller General Chair of ICSE 2001 for the use there
• In discussion with Conference Services of the National Research Council for the use of all conferences organized by the Council

A very careful consideration was given to:

• Access Controls:. A flexible, comprehensive mechanism of various permissions was implemented to allow: (1) authors to share access to joint submissions without letting others see or even know about them; (2) referees to enter and edit their reports in the knowledge that authors cannot see them even if they are referees or PC Members themselves; and (3) PC Members to read and contribute to each others work.
• Data Integrity: A monotonic database (no deletions) is combined with full transaction logging viewable by the record's creator and all of those with higher levels of permission. This facilitates: (1) ability to track and back out of undesired changes made in full view (security camera model); (2) ability to modify the database schema and correct bugs without loss of data by replaying the logs; and (3) the backup of the database by saving logs.
• Usability: A generally open model is provided that allows more operations to be performed than needed in order to facilitate sharing and co-operation. This is combined with the access control and security camera mechanisms to protect the integrity of the system, the database, and the refereeing process. Attention has also been paid to ensure that the flow between forms is sensible and easy-to-use by concentrating on user-oriented tasks. This is especially true for the reviewers, the users of the system with the least investment in the successful outcome.
• Performance: The use of modPerl combined with a careful design of the scripts overcomes many of the delays caused by the CGI model. This allows us to expire almost all pages with data content immediately resulting in more consistent behaviour from the user's point of view.

Being a Software Engineering Group, we put much effort into making the software flexible, maintainable and secure. The Group is involved in COTS and Configuration Management research and WitanWeb was subjected to a rigorous development process.

[Top of Page ]

http://www.cs.washington.edu/homes/mernst/conference/resources.html lists a variety of resources, including some collections of scripts.

SIGs who have expressed interest in conference management software:

• SIGACT - Jeff Vitter  jsv@cs.duke.edu
• SIGCOMM - Sally Floyd  floyd@ee.lbl.gov
• SIGCPR - Al Lederer <Lederer@UKCC.UKY.EDU>
• SIGCHI - Wayne Graves <graves@acm.org>, Kevin Scofield   <kevinsch@microsoft.com>, Gerrit van der Veer <gerrit@cs.vu.nl>, Michael Atwood <Michael.Atwood@cis.drexel.edu>
• SIGCHI has been using a Web-based reviewing-management application  that was written by Lotus Research. They are now licensing it out through their Lotus Business Partners office, which allows third parties to license it and then host it for organizations such asourselves.
• SIGDA - Steve Levitan levitan@ee.pitt.edu
• SIGGRAPH - Conference Advisory Group <cag@siggraph.org>
• SIGMOD - Michael L. Brodie <brodie@gte.com>, Meral Ozsoyoglu <ozsoy@alpha.ces.cwru.edu>
• SIGPLAN - Ron Cytron  cytron@cs.wustl.edu
• SIGSOFT - David Garlan  <garlan@cs.cmu.edu>, Alex Wolf <alw@cs.colorado.edu>
• VLDB - John Mylopoulos <jm@cs.toronto.edu>

Other questions:

• Administration tools (e.g., overview of the review process)
• Running the PC meeting. A lot of work goes into making that go smoothly - even when the meeting isn't "distributed"
• The amount of time it takes to personalize a given system to a conference - meaning how much coding is required.  This is the thing that really annoyed Program Chairs in the past.
• what bookkeeping aspects are supported, e.g., payment of particiants, overview over balance, etc.
• posibility to identify "critical" papers and/or PC members (i.e., missing reviews)
• overview of online ballot/voting process
• support for creating report(s) (both ad hoc and predefined)
• support for "blind" reviews
• are review results visible to other PC members during the review phase
• possibility to exclude PC members from voting/discussing several papers, e.g., if a PC member is a co-author