SIG Governing Board Minutes: October 23, 2020
*list is not all-inclusive
Natalie Enright Jerger
Kalyan S. Perumalla
Duration: 3 Hours
Meeting starts at 7:06
Jonathan Aldrich (liaison to the ACM PUBS board) + Scott Delman
Jonathan speaks from 11:23 – 20:56
- Double Blind Reviewing
Should ACM mandate some form of/partial Double Blind Reviewing?
Scott Delman: The pubs board is very clear that all authors including conferences and journal authors have the right to post their pre-prints at any stage in the process
Helena Mentis: Full transparency of reviewers so that bias and bad behavior cannot be hid behind anonymity. The movement right now proposes that a review should be signed to indicate that a reviewer stands by their review.
Shan Lu: Listing all reviewer names (tied to a paper) without linking the review to the reviewer’s name may help reduce ethical violations
Samir Khuller: Worries that reviews will become sugar coated if we require them to be signed, particularly in the case of junior employees reviewing the work of their superiors who may need their assistance in the future
Srinivas Akru: Reviewer anonymity should not be compromised. Compromised anonymity may lead to sugar coated reviews in the context of a junior position holder reviewing their senior’s work and in the context of people in power (those on funding panels) deliberating with a bias against people who rejected their work in the past.
Peter Brusilovsky: We should not have a mandate but instead a best practices in regards to double blind
Chris Brown: A double blind mandate policy would be problematic, and it may end the relationship between SIGSAM and one of our sponsored/in co-op conferences (ISSAC) which so happens to be an independent entity with their own bylaws
Jeanna Matthews: would like to see a Best Practices White Paper considering the differences in opinion being shared specifically in regard to marginalized groups. Proposed for more experimentation with DBR before adopting a new movement or mandate.
Adam Bargteil: Also a proponent for a detailed Best Practices White Paper that can be shared with Program Chairs
Shan Lu: There is a difference between discussion-oriented workshops and conferences. People are more interested in discussing a paper from a well-known company like Google as a workshop in comparison to a lesser known company, hence double blind is not practiced in this context by SIGOPS where discussion is prioritized over anonymity.
Sujata Banerjee: SIGCOMM’s operational systems track papers - it made no sense to anonymize the company name, so we still anonymized authors (edge case example similar to what Shan Lu discussed in regard to SIGOPS workshop)
Marco Gruteser: Supportive of sending a strong signal to reduce bias in review processes. An option to handling the issue of DBR would be to strongly suggest double blind but allowing Conferences the option of submitting an ultimate plan for reducing bias in their peer review processes.
Sanmay Das: scarcity of reasonable reviewers; are reviewers anonymous to each other? It would be great to have best practices to know what other folks are doing.
Kalyan S. Perumalla: wants guidance regarding transition strategies from non-double blind to double blind reviewing in conferences
Sharon Hu: What is the real reason behind a decline in review quality? Perhaps this is something the PUBS board should investigate. Sharon fully supports the notion of general best practices from ACM regarding DBR.
Juliana Freire: How do we reduce bias while simultaneously allowing people to do significant incremental work? Not being able to refer to prior work in the context of DBR can create issues for authors who have had their work rejected because they did not cite their own work.
Babak Falsafi: It would be good to discuss the journal first model in best practices
High Level Takeaways as per Jonathan Aldrich
- A lot of support for Double-Blind in general
- A lot of requests for a Best practices white paper & more information about how to implement it effectively
- A need for flexibility, we would not want to have a strict mandate because some of these conferences are coordinating, there is also a desire to experiment with some other models that may perhaps help marginalized communities in different ways
2. Peer Review Report
Jonathan speaks from 59:34 – 1:04:00
Would creating an ACM Conference-wide standard timeline for completing reviews (3 weeks, for example) be welcomed? Potential changes to the minimum number of reviews required to make publication decisions for journal and conference submissions (from 3 to 2).
Jens Palsberg: There is some experimentation with SIGPLAN conferences. For example, a paper was accepted with two strong reviews in the first round, whereas another was rejected that had two strong negative reviews. There was not a need for a third review in these circumstances.
Shan Lu: It would not be ideal to have a unified timeline across conferences considering the paper review workload varies across conferences. Perhaps eventually we will shift to a model in which papers are rejected with two reviews, but for now that seems like a difficult call to make.
Franz Rothlauf: There has been an increase in submissions with a simultaneous decrease in quality. SIGEVO is not pushing hard on decreasing the timeline regarding video submissions considering it ended up reducing the amount of video submissions received.
Sharon Hu: We should separate the discussion when it comes to journals and conferences. 3-week review timeline is not workable and should not be mandated. How can we create a mechanism to get more reviewers to sign up? Do we need so many conference and journal publications? Can we think about a model in which conferences are more about socializing?
Jian Pei: The conference publication acceptance process should be made easier, which is something we will implement in our KDD ’21 conference to provide a more inclusive community. Services like arxiv are a threat to the conference and journal publication business model, hence the need to be more accepting regarding publications.
Shan Lu: ASPLOS is doing an experiment this year in which they asked authors to submit a 2 page extended abstract and a 10 page full paper with the 2 page extended abstract being rejected with 5 reviews in the first round as opposed to the 10 page paper being rejected with 2 reviews in the first round. Advocating for the integration of TPMS with HotCRP
Adam Bargteil: The trend regarding publications seems to be moving away from conferences and more towards journals. SIGGRAPH is trying to come up with a procedure for reviewer continuity from conferences to journals.
Sharon Hu: Proponent for getting author’s feedback regarding their reviews and or rejections and giving them a booming voice in general. Would like to see ACM’s guidance around this.
3. Orcid (a way of identifying authors)
Jonathan speaks from 1:23:30 – 1:28:20
Jens Palsberg: Strongly in favor of open access because it helps eliminate ambiguity surrounding author identity
Juliana Freire: In favor of Orcid because it would help with conflict of interest detection
Helena Mentis: Can authors use pseudonyms to protect themselves against harm?
Scott Delman: Orcid does not prevent you from using a pseudonym to register ^. This mandate will not be burdensome to authors as we are not mandating them to take ownership of their bibliography, which is the time-consuming portion of this process. Scott along with a few other staff members will provide resources for anyone that needs assistance with Orcid.
Laurie Fox: We are missing networking on our f2f conferences, in support of Sharon Hu’s viewpoint. Worries that requirements such as DBR and Orcid will result in fewer conference submissions.
Peter Brusilovsky: A thoughtful implementation of Orcid will make a difference in terms of saving time as opposed to being burdensome
Fred Niederman: Orcid implementation should put the authors’ interests first, otherwise it may become an additional burden resulting in a lower amount of author participation. How can we make Orcid collection seamless by integrating it with pre-existing processes that authors are already participating in as part of their submission process?
High Level Takeaways as per Jonathan
Making sure this stays very easy for Authors particularly Authors who do not publish very much
This disproportionately helps groups like women and transgender people who are more likely to have multiple names
There is also a very positive diversity related reason to go for this in addition to all the other benefits
Conferences: Virtual & Hybrid in 2021
Jens speaks from 1:42:00 – 1:53:00 (2 case studies ICFP 2020 + MODPODS 2020)
Business Model? Cost of getting help? What are people willing to pay?
ICFP 2020 survey of 412 attendees indicates that students are willing to pay between $10-50 for registration, whereas Academics are willing to pay between $50-200 for registration
High Level Takeaways as per Jens
- People are willing to pay
- The amount charged for registration (in context of the above-mentioned case studies) lines up with attendees’ expectations regarding registration costs
How does this match up with your experiences of virtual conferences over the last 6 months? What are your expectations for conferences coming up as it relates to the Business Model?
Juliana Freire: Students were not charged registration fees for MODPODS, only conference & workshop authors were charged registration fees. This resulted in a surplus of funds and increased attendee participation both numerically and geographically.
Laurie Fox: How is sponsorship staying so high? (in reference to case studies)
Most of our sponsors are in the same financial boat our colleges are in and they are not wanting to continue support which really impacts our revenue.
Benjamin Pierce: (In response to Laurie Fox) We should not trust numbers we are seeing for 2021 because a lot of those sponsorship commitments were locked in before things became erratic.
2. 2021 Conferences
Renee McCauley: Are ’21 conferences securing sponsorships at the same level they have in the past?
Juliana Freire: We need to be conservative in the future because sponsorships are likely to go down.
Sharon Hu: Virtual exhibit for large SIGDA conference resulted in a big financial loss this year. How are other SIGS handling this type of problem? What are some good practices SIG leaders can share with conferences to help achieve success in hybrid events?
Adam Bargteil: SIGGRAPH will most likely go purely virtual but leave the door open for some in person events in LA (drive in screening - Computer Animation Festival)
Franz Rothlauf: Planning to run GECCO as a hybrid event in Summer ’21 with 100-200 people onsite with the rest online. Will not rely on attendees for technical equipment in contrast to traditional physical events and they will supply the equipment for all in person attendees. This year’s GECCO attendees who attended the conference for free were required to join SIGEVO which the SIG determined helped increase attendance numbers and engagement throughout the duration of the conference.
Jian Pei: KDD’21 will either be a small in person conference or a large virtual conference, this is due to the difficulty of running a hybrid event. A decision has not yet been made as to which direction SIGKDD will go in, however they are eager to learn best practices and from the experience of other SIGS.
Jens Palsberg: in complete alignment with Jian Pei’s above comment
Juliana Freire: possible strategy would be to set a maximum number of attendees in combination with requiring early registration so that people are committed in terms of their attendance
Jens Palsberg: What are your SIGS thinking the interaction type (F2F, Virtual, Hybrid) will be for your May – Aug conferences in ‘21?
Adam Bargteil: Going forward we will probably have pre-recorded talks you can watch ahead of the conference, with the in-person portion focused on networking & roundtable discussions
What is a good platform for virtual conferences?
Adam Bargteil: Zoom for Q&A sessions, Discord for background Q&A & asynchronous conversations, & YouTube for posting talk videos
Sharon Hu: Underline, I have been happy with pre-conference preparations. They call themselves white glove services.
Shari Trewin: We need to keep accessibility in mind when selecting conference platforms. Some platforms do a better job at providing accessibility features in the context of being more socially inclusive.
Jeff Jortner: Advocated for Bizabbo due to integration capabilities, does not like bouncing between multiple platforms with each platform serving a different purpose. Finding a platform that supports better dynamic social interaction and inclusivity will be a challenge worth solving.
Diversity & Inclusion
Natalie Enright Jerger speaks from 02:27:23 – 02:37:20 then paused for questions resumes at 2:39:40 – 2:48:32
- Would like to know demographics regarding authorship
- New implementation coming soon that will decouple collection of data from submission system (authors have the option of voluntarily submitting their demographic data via an ACM website)
- D&I council can offer support regarding industry sponsors inquiring about conference D&I initiatives
- Cares is a SIG-level program to aid in the reporting of harassment at ACM events, Cares has been expanded to include ethics/publication issues
- Words matter (Blacklist/whitelist) Exclusionary, culturally specific
- Formed Standing Committee on Systemic Change to address issues of systemic racism addressed in letter of concern from Black ACM members
Peter Brusilovsky: We need to think about how we can increase geographic participation without excessively charging authors who are essentially compensating for attendees that are not paying any registration fees and who may not be able to afford registration fees in the first place.
Also, we need to look down the line and ensure we encourage diverse students to pursue higher education, prepare them for graduate school and as a result the up the line positions will have a greater chance of being filled by qualified and diverse professionals.
Juliana Freire: SIGMOD started a diversity and inclusion initiative to figure out ways to include more people in their community
End of meeting