Written by: Charlie Schweik & Mohammed Zia · Posted on: May 08, 2017
FOSS4G Academic Track and Review Process
With a changing world tide, especially in science and technology, we have to listen to and involve as many communities as possible to create a fit-for-all geo-solution. The world problems have evolved and so should our approach.
There is an utmost requirement for the Open-Source community to bring in scientists, researchers, developers, end users, and almost all tiers of the geospatial paradigm to confront and answer current geo-challenges plaguing our planet Earth and humanity. The FOSS4G 2017 Conference will be an important bridge between academia and industry that leads to new ideas and innovations.
To highlight the top quality academic work that is motivated by and is carried out with FOSS4G data and software, the Conference solicited academic papers in association with oral presentations. The FOSS4G 2017 Academic Committee consisted of 21 individuals having a wide range of expertise in the development and application of FOSS4G technologies, and with academic backgrounds that generally included refereeing papers submitted to journals. They were charged with reviewing the academic abstracts and selecting the best of them for further evaluation of the written article. These papers will be published in the Conference Proceedings, but the best of the papers will also be promoted for consideration for publication in an internationally known GIS journal (yet to be confirmed).
There were 49 academic abstracts amongst the 416 all-conference submissions, and after double-blind review the Academic Committee selected 30 for acceptance as papers. Another 14 were considered worthy for inclusion as posters (which will also be published in the Proceedings). The remaining abstracts were rejected for not meeting the Committee’s agreed upon standards (one was a duplicate).
The FOSS4G 2017 Academic Committee developed a set of criteria for assessing the academic abstracts, and three committee members were assigned to review each one based on their areas of expertise, with no identifying information included.
The reviewers evaluated their assigned abstracts along five components from 1 (worst) to 5 (best):
- Technical Quality
- Relevance to FOSS
- Relevance to Geography
- Academic Interest
The reviews for each of these components were averaged together for each abstract as a tertiary guide for review of abstracts at the margins. The resulting abstract components were also averaged together with a uniform weight to provide a single Component Composite, as a secondary guide.
The reviewers also provided a summary Final Recommendation, which was averaged together for each abstract using a linear numeric scale:
- Strong Accept (5)
- Borderline Accept (4)
- Accept as Poster (3)
- Borderline Reject (2)
- Strong Reject (1)
As might be expected, there is a high correlation between the Final Recommendation and the Component Composite, 0.79.
In the Acceptance List, the abstracts were sorted by their Final Recommendation Average and converted back to the text recommendations above; + or – were added to each if they were an integer ± 1/3 (indicating non-consensus), with the following results:
Recommendation Number Decision Strong Accept 5 Paper Strong Accept - 10 Paper Borderline Accept + 5 Paper Borderline Accept 7 Paper Borderline Accept - 7 3 Paper, 4 Poster Accept as Poster + 8 Poster Accept as Poster 1 Poster Accept as Poster - 1 Reject Borderline Reject + 2 1 Poster, 1 Reject Borderline Reject 1 Reject Strong Reject 2 Reject
There were 27 abstracts that were “Borderline Accept” or better, meaning a consensus on acceptance. The Co-Chairs also looked at the next level of 7 that were “Borderline Accept –”, and using the component evaluations and reviewer comments noted a distinction that led to three more being accepted for paper submission, for 30 all together.
The next group of abstracts were considered for posters in the same way, with 9 that were “Accept as Poster” or better. Consideration of the components and reviewer comments led to the acceptance of one “Borderline Reject +” as a poster, and rejecting one “Accept as Poster –”, along with the remaining five abstracts.
The Academic submissions were also reviewed by the Program Committee and were also voted on by the Community, sometimes with different results reflecting the perspectives and interests of these more diverse groups. When a paper was accepted for a regular oral presentation, the academic submitters were notified of this and were provided with the opportunity to withdraw from the Academic Program with its requirement for an academic paper or poster.
We deeply appreciate all those who took the time to submit an abstract for consideration by the Academic Committee, and all of the reviewers for their time and effort for arranging and undertaking this review. Of course, no process is free from false negatives and the process we describe above is no exception. We understand that a few abstract submitters will be upset with their result, but we hope the above explains that we did our best to run a selection process that was unbiased, fair, used peer-review expertise, and was systematic in its selection. This was no easy task on our end, and we do believe that the FOSS4G community will support the decisions at this stage, as well in later stages when we review the submitted, accepted papers in June.
Please feel free to get back to us for any kind of assistance. We are always at your disposal.
Prof. Dr. Charlie M. Schweik Chair of FOSS4G Boston 2017 Academic Committee Email: firstname.lastname@example.org
Mohammed Zia Co-Chair of FOSS4G Boston 2017 Academic Committee Email: email@example.com
Thanks again, you all, for making FOSS4G 2017 a reality. We hope to see you in Boston, U.S.A. in August! (Register now)
Written by: Michael Terner · Posted on: April 10, 2017
An embarrassment of riches, and difficult cuts
Full disclosure, I write this blog not just as the FOSS4G Boston 2017 Conference Chairman and a member of our program committee but also as a member of two previous FOSS4G North America program committees and as someone who has had an abstract rejected from a FOSS4G event (and several others accepted). I also write this from my own perspective and I do not speak on behalf of the entire program committee.
There’s an old saying that goes “careful of what you wish for” and I think our Program Committee is feeling that strongly. On the Friday four days prior to the Call for Presentations deadline, our committee knew that 90 abstracts had been submitted, and we wished mightily for many more (knowing we had room for over 200 talks). Well our wish came true and over those last four days we received 336 more abstracts for a total of 416 (and we received several requests to consider abstracts from people who missed the deadline). This was great news, but the “careful what you wish for” part is being felt in two large ways:
- We have a lot more work to do. Spending 5 minutes per abstract 416 times adds up to over 34 hours for each reviewer. And, reading and scoring is just the start of the selection process.
- After reading just the first several dozen it was clear that the general quality of abstracts was very high. The number of rooms we have and the general program that has been laid out tell us that somewhere between 175 - 200 abstracts will need to be rejected.
As such, we anticipate that the decision making process is going to be arduous, complicated and emotionally difficult knowing that many papers won’t be accepted. We also know that a lot of work went into creating abstracts and that in some cases having an abstract accepted impacts one’s ability to attend the conference. Given these realities and in the spirit of free and open communication I thought it might be useful to share some of the things the program committee thought about in preparing our Call for Presentations and also how we are entering the deliberation phase of the process.
First, and most importantly, the Boston team has a vision for the kind of program that we believe will underpin a great event and attract the largest and widest audience to Boston in August. We also want an event that will help to strengthen the OSGeo community. We laid that vision out in our Call for Presentations (which has now been taken offline, but can be found here, and some of that outlook remains on the Program Page of the website.
The basics of what we are looking for:
- A comprehensive program that is attractive to developers and users of Open Source. A program with content for both advanced, long-time users as well as novices and those just “checking out” FOSS4G software.
- Content that represents the wide variety of uses of FOSS4G software from government to private industry; from transportation to economic development to emergency response.
- A diversity of voices, from experienced FOSS4G presenters to those new to the community, and including a cross section of gender, geography, and ethnicity.
- Representation from the emerging business ecosystem that surrounds the FOSS4G community. How are people creating businesses that employ or support FOSS4G?
Beyond finding content to help us realize our vision for the conference there are other important aspects that need to be considered in making the individual acceptance decisions.
- Quality matters The writing and descriptions in the abstracts are extremely important in helping us imagine what the presentation will be like.
- Managing fairness of quantity matters There are a large number of individuals and organizations who submitted more than one abstract (i.e., over 60 people submitted >1 abstract, and 16 organizations submitted four, or more abstracts). While there will undoubtedly be some individuals who have >1 abstract accepted we are mindful that it is challenging to reject someone’s single entry while accepting multiple entries from others.
- Respecting community voting matters We have asked the OSGeo Community to vote for what you want to see. And, we will respect that and approximately 20% of the program will be chosen directly by the top community voting results. And, for all other decisions, the community vote will be considered as one of several important variables. That said, we are also mindful that the community vote is not necessarily a representative sample of those who will attend the conference.
In short, we have a fantastic group of Program Committee volunteers who are working extremely hard on this process and recognize the gravity of the decisions we are making. We are all human and fallible and we will do the best we possibly can. It is also fair to recognize that it will be easy to second guess our decisions and that invariably we may make some mistakes. We ask that this community be empathetic and patient with us and the task ahead.
Here is what comes next:
- Our committee has committed to trying to complete all first pass reviews of all 416 abstracts by Monday, April 10th (i.e., the same day as community voting closes).
- Our committee is working to make some early acceptance announcements, potentially before community voting closes as there is already some strong agreement on several abstracts. We believe this is helpful as it allows the authors of accepted abstracts the opportunity to begin their next level of planning to attend. And, at the same time it helps others get a better sense of the emerging program and have more information to base their own decisions on attending, and potentially making earlier travel arrangements and registration. Once the community voting is closed, we will try to make those early announcements quickly, and no later than Friday, April 14.
- We will then continue to make incremental acceptance announcements on a regular basis to continue revealing the program as early as we can.
- The last round of final announcements will be made on Monday, May 1.
Thanks to the entire FOSS4G community for your support.