Big Data Analytics Lab

[1] Alnur Ali and Marina Meilă. 2012. Experiments with Kemeny ranking: What works when? Mathematical Social Sciences 64, 1 (2012), 28–40.

[2] Sihem Amer-Yahia, Shady Elbassuoni, Behrooz Omidvar-Tehrani, Ria Mae Borromeo, and Mehrdad Farokhnejad. 2019. Grouptravel: Customizing travel packages for groups. In 22nd International Conference on Extending Database Technology (EDBT).

[3] Sihem Amer-Yahia, Senjuti Basu Roy, Ashish Chawlat, Gautam Das, and Cong Yu. 2009. Group recommendation: Semantics and efciency. Proceedings of the VLDB Endowment 2, 1 (2009), 754–765.


[4] Kenneth J Arrow. 1950. A difculty in the concept of social welfare. Journal of political economy 58, 4 (1950), 328–346.


[5] Kenneth J Arrow, Amartya Sen, and Kotaro Suzumura. 2010. Handbook of social choice and welfare. Elsevier.

[6] Anastasios Arvanitis and Georgia Koutrika. 2012. Towards preference-aware relational databases. In 2012 IEEE 28th International Conference on Data Engineering. IEEE, 426–437.

[7] Haris Aziz and Nisarg Shah. 2021. Participatory budgeting: Models and approaches. In Pathways Between Social Science and Computational Social Science.Springer, 215–236.

[8] John Bartholdi and James Orlin. 1991. Single transferable vote resists strategic voting. Social Choice and Welfare 8 (01 1991), 341–354. https://doi.org/10.1007/BF00183045

[9] Sanjoy K Baruah, Neil K Cohen, C Greg Plaxton, and Donald A Varvel. 1996. Proportionate progress: A notion of fairness in resource allocation. Algorithmica 15, 6 (1996), 600–625.

[10] Sanjoy K Baruah, Johannes E Gehrk, C Greg Plaxton, Ion Stoica, Hussein AbdelWahab, and Kevin Jefay. 1997. Fair on-line scheduling of a dynamic set of tasks on a single resource. Inform. Process. Lett. 64, 1 (1997), 43–51.

[11] Senjuti Basu Roy, Laks VS Lakshmanan, and Rui Liu. 2015. From group recommendations to group formation. In Proceedings of the 2015 ACM SIGMOD international conference on management of data. 1603–1616.

[12] Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. 2019. Fairness in recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2212–2220.

[13] Ronen Brafman and Carmel Domshlak. 2004. Database preference queries revisited. Technical Report. Cornell University.

[14] Steven Brams and Peter C. Fishburn. 2007. Approval voting.

[15] Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D. Procaccia. 2016. Handbook of computational social choice. Cambridge University Press.

[16] Ioannis Caragiannis, Jason A Covey, Michal Feldman, Christopher M Homan, Christos Kaklamanis, Nikos Karanikolas, Ariel D Procaccia, and Jefrey S Rosenschein. 2012. On the approximability of Dodgson and Young elections. Artifcial Intelligence 187 (2012), 31–51.

[17] L Elisa Celis, Damian Straszak, and Nisheeth K Vishnoi. 2017. Ranking with fairness constraints. arXiv preprint arXiv:1704.06840 (2017).

[18] Jan Chomicki. 2003. Preference formulas in relational queries. ACM Transactions on Database Systems (TODS) 28, 4 (2003), 427–466.

[19] Edith Elkind et al. 2017. What do multiwinner voting rules do? An experiment over the two-dimensional Euclidean domain. In AAAI ’17. 494–501.

[20] Ulle Endriss. 2017. Trends in computational social choice. AI Access.

[21] Piotr Faliszewski, Piotr Skowron, Arkadii Slinko, and Nimrod Talmon. 2017. Multiwinner voting: A new challenge for social choice theory. In Trends in computational social choice, Ulle Endriss (Ed.). AI Access, 27–47.

[22] Allan Gibbard. 1973. Manipulation of voting schemes: a general result. Econometrica: journal of the Econometric Society (1973), 587–601.

[23] Lei Guo, Hongzhi Yin, Qinyong Wang, Bin Cui, Zi Huang, and Lizhen Cui. 2020. Group recommendation with latent voting mechanism. In 2020 IEEE 36th International Conference on Data Engineering (ICDE). IEEE, 121–132.

[24] Edith Hemaspaandra, Holger Spakowski, and Jörg Vogel. 2005. The complexity of Kemeny elections. Theoretical Computer Science 349, 3 (2005), 382–391.

[25] Zhenhua Huang et al. 2020. Social group recommendation with TrAdaBoost. TCSS (2020).

[26] Md Mouinul Islam, Mahsa Asadi, and Senjuti Basu Roy. 2023. Equitable Top-k Results for Long Tail Data. Proceedings of the ACM on Management of Data 1, 4 (2023), 1–24.

[27] Md Mouinul Islam, Soroush Vahidi, Baruch Schieber, and Senjuti Basuroy. 2024. Promoting Fairness and Priority in �-Winners Selection Using IRV. In KDD. ACM. https://doi.org/10.1145/3637528.3671735

[28] Md. Mouinul Islam, Dong Wei, Baruch Schieber, and Senjuti Basu Roy. 2022. Satisfying complex top-k fairness constraints by preference substitutions. Proc. VLDB Endow. 16, 2 (oct 2022), 317—-329.

[29] Georgia Koutrika and Yannis Ioannidis. 2005. Personalized queries under a generalized preference model. In 21st International Conference on Data Engineering (ICDE’05). IEEE, 841–852.

[30] Caitlin Kuhlman and Elke Rundensteiner. 2020. Rank aggregation algorithms for fair consensus. Proceedings of the VLDB Endowment 13, 12 (2020).

[31] Thomas R. Magrino, Ronald L. Rivest, Emily Shen, and David Wagner. 2011. Computing the margin of victory in IRV elections. In Proceedings of the 2011 Conference on Electronic Voting Technology/Workshop on Trustworthy Elections (San Francisco, CA) (EVT/WOTE’11). USENIX Association, USA, 4.

[32] Kenneth O May. 1952. A set of independent necessary and sufcient conditions for simple majority decision. Econometrica: Journal of the Econometric Society (1952), 680–684.

[33] Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. 2022. Fairness in rankings and recommendations: an overview. The VLDB Journal (2022), 1–28.

[34] Senjuti Basu Roy. 2022. Returning top-k: Preference aggregation or sortition, or is there a better middle ground. SIGMOD Blog (2022).

[35] Senjuti Basu Roy. 2024. Fairness and Robustness in Answering Preference Queries. Data Engineering (2024), 36.

[36] Donald G. Saari. 2006. Which is better: the Condorcet or Borda winner? Social Choice and Welfare 26, 1 (2006), 107.

[37] Anand Sarwate, Stephen Checkoway, and Hovav Shacham. 2013. Risk-Limiting Audits and the Margin of Victory in Nonplurality Elections. Statistics, Politics, and Policy 4, 1 (Jan. 2013), 29–64.

[38] Mark Allen Satterthwaite. 1975. Strategy-proofness and Arrow’s conditions: Existence and correspondence theorems for voting procedures and social welfare functions. Journal of economic theory 10, 2 (1975), 187–217.

[39] Ashudeep Singh, David Kempe, and Thorsten Joachims. 2021. Fairness in ranking under uncertainty. Advances in Neural Information Processing Systems 34 (2021), 11896–11908.

[40] Piotr Skowron, Arkadii Slinko, Stanisław Szufa, and Nimrod Talmon. 2020. Participatory budgeting with cumulative votes. arXiv preprint arXiv:2009.02690 (2020).

[41] Julia Stoyanovich, Meike Zehlike, and Ke Yang. 2023. Fairness in Ranking: From Values to Technical Choices and Back. In Companion of the 2023 International Conference on Management of Data. 7–12.

[42] Nimrod Talmon and Piotr Faliszewski. 2019. A framework for approval-based budgeting methods. In AAAI ’17. 2181–2188.

[43] Nicolaus Tideman. 1995. The single transferable vote. Journal of Economic Perspectives 9, 1 (1995), 27–38.

[44] Robert Tijdeman. 1980. The chairman assignment problem. Discrete Mathematics 32, 3 (1980), 323–330.

[45] Dong Wei, Md Mouinul Islam, Baruch Schieber, and Senjuti Basu Roy. 2022. Rank Aggregation with Proportionate Fairness. In Proceedings of the 2022 International Conference on Management of Data (Philadelphia, PA, USA) (SIGMOD’22). Association for Computing Machinery, New York, NY, USA, 262––275.

[46] Peyton Young. 1995. Optimal voting rules. Journal of Economic Perspectives 9, 1 (1995), 51–64.

[47] Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza-Yates. 2017. FA*IR: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1569–1578.

[48] Meike Zehlike, Ke Yang, and Julia Stoyanovich. 2021. Fairness in ranking: A survey. arXiv preprint arXiv:2103.14000 (2021).

Senjuti Basu Roy Baruch Schieber Nimrod Talmon
senjutib@njit.edu sbar@njit.edu talmonn@bgu.ac.il
New Jersey Institute of Technology New Jersey Institute of Technology Ben-Gurion University

 

Slides  |   References  |   PDF

 

Abstract

Given a large number (notationally m) of users’ (members or voters) preferences as inputs over a large number of items or candidates (notationally m), preference queries leverage different preference aggregation methods to aggregate individual preferences in a systematic manner and come up with a single output (either a complete order or top-k, ordered or unordered) that is most representative of the users’ preferences. The goal of this tutorial is to adapt different preference aggregation methods from social choice theories, summarize how existing research has handled fairness over these methods, identify their limitations, and outline new research directions.

PART I: Preference aggregation Method

In this part, we describe the basic model in computational social choice—voting. In voting, preferences are first elicited from an agent community, which are aggregated in the subsequent step. We formally define election, voting rule, input and output formats, preference aggregation, and analysis of the approach.

PART II: Fairness in answering preference queries - existing research

We model fairness by protected attributes. Each item/candidate is associated a set protected attributes. As an example, seniority level is a multi-valued protected attribute with three possible values Junior, Mid career, Senior, while gender is commonly a binary protected attribute with two values male and female. Building upon this, we summarize existing research about fairness in answering preference queries in several key aspects:

  • ensuring fairness;
  • multi protected attributes;
  • producing a fair outcome.

PART III: Future research directions

We focus on three major aspects:

  • new preference aggregation methods;
  • alternative models to enable fair outcomes;
  • efficient solution design.

Presenters

Senjuti Basu Roy is the Panasonic Chair in Sustainability and an Associate Professor in the Department of Computer Science at the New Jersey Institute of Technology. Her research focus lies at the intersection of data management, data exploration, and AI, especially enabling human-machine analytics in scale. Senjuti has published more than 85 research papers in high impact data management and data mining conferences and journals. She has served as the tutorial co-chair of VLDB 2023.

Baruch Schieber is a Professor in the Department of Computer Science, NJ Institute of Technology. Before joining NJIT, Baruch was a Distinguished Research Staff Member in IBM Research. His research interests are in theoretical computer science, including optimization under uncertainty, algorithms, mathematical programming, and high-performance computing. Baruch received his PhD in Computer Science from Tel Aviv University in 1987. He published more than 150 papers in scientific journals and conferences.

Nimrod Talmon is an Assistant Professor within the Industrial Engineering and Management Department, Ben-Gurion University, Israel. Before joining Ben-Gurion University, he was a Postdoctoral Fellow with the Computer Science and Applied Mathematics Department, Weizmann Institute of Science, Israel and received his Ph.D. degree in computer science from TU Berlin. His research interests include artificial intelligence, game theory, computational social choice, social networks, and combinatorial optimization. He has Erdos number 3, Sabbath number 7, and Bacon number 6.

CNRS, Univ. Grenoble Alpes, New Jersey Institute of Technology
Sihem AmerYahia
sihem.ameryahia@cnrs.fr
Senjuti Basu Roy
senjutib@njit.edu

Time: August 28, 4:00  - 5:30 pm, Location: Avalon

Slides     |     References

Abstract

The goal of this tutorial is to make the audience aware of various discipline-specific research activities that could be characterized to be part of online labor markets and advocate for a unified framework that is interdisciplinary in nature and requires convergence of different research disciplines. We will discuss how such a framework could bring transformative effect on the nexus of humans, technology, and the future of work.

PART I: Applications

In this part, we will describe dierent applications that tap into on-line labor markets. The applications range from free-lancing, crowd-sourcing, citizen science, as well as flash organizations. We will characterize these applications by describing the nature of the type of work/business and workers, thereby highlighting the desirable properties that each must meet.

PART II: Existing Approaches

Computational DB and ML
Non-Computational Social Science, Psychology
Data and Problem Modeling, Solutions, and Impact

This piece of the tutorial will revisit existing efforts from computational and non-computational communities and summarize them along three dimensions : studied problems, propoposed solutions, and their impacts.

PART III: Toward a Unified Framework

Challenges and opportunities in unifying the design of online labor markets

This part is primarily forward looking and aims to discuss the challenges and opportunities that arise from bringing together empirical and computational approaches to unify the design of future online labor markets.

Presenters

Sihem Amer-Yahia: is a CNRS Research Director at the University of Grenoble Alpes where she leads the SLIDE team. Her interests are at in large-scale data management. Before joining CNRS, she was Principal Scientist at the Qatar Computing Research Institute, Senior Scientist at Yahoo! Research and at&t Labs. Sihem has served on the SIGMOD Executive Board, is a member of the VLDB and the EDBT Endowments. She is the Editor-in-Chief of the VLDB Journal and an Associate Editor for Transctions in Data Science. She was PC chair of VLDB 2018. Sihem received her Ph.D. in Computer Science from Paris-Orsay and INRIA in 1999, and her Dipl^ome d'Ingenieur from INI, Algeria. Sihem is co-organizing a Shonan Meeting on the topic of Human-in-the-loop Big Data and AI: Connecting Theories and Practices for a Better Future of Work.

Senjuti Basu Roy: is an Assistant Professor at the New Jersey Institute of Technology. Her broader research interests lie in the area of data and content management with the focus on designing principled algorithms for ``human-in-the-loop'' systems. She was the PC Co-chair of SIGMOD 2018 mentorship track, VLDB 2018 PhD Workshop program, and the IEEE Workshop on Human-in-the-loop Methods and Human Machine Collaboration in BigData (IEEE HMData2017, 2018, 2019) (co-located with IEEE Big data). She has organized  NSF workshop on converging human and technological perspectives in crowd-sourcing research and will be co-organizing a Shonan Meeting on the topic of Human-in-the-loop Big Data and AI: Connecting Theories and Practices for a Better Future of Work.

Please contact our directors via e-mail:  

senjutib@njit.edu
bdal@njit.edu 

Big Data Analytics Lab
University Heights
Guttenberg Information Technologies Center (GITC) Suite 4111
​Newark, New Jersey 07102

✨ New Update [April 2025] -- A new project, in collaboration with Boston Fusion Corporation, sponsored by Office of Naval Research, titled "RUBICON: Form Curation While Completion", has been awarded for 4 years!
 
✨ New Update [April 2025] -- Prof. Senjuti Basu Roy is serving as a Research PC-Co Chair of CIKM 2025, the 34th ACM International Conference on Information and Knowledge Management. Please consider making a submission!
 
✨ New Update [March 2025] -- Phase-2 of a project, sponsored by Office of Naval Research, titled "Predictive Analytics on Ship Scheduling", has resumed and will go for another 3 years!
 

✨ May 2024 -- Congratulations to Sepideh Nikookar for winning the Joseph Leung Award for the Best CS PhD Dissertation (academic year 2023-2024)!

✨ May 2024 -- Gerald White presented a poster at North East Database Day on May 23 titled, "Predict Days of Maintenance Delay (DoMD) of US Naval Vessels: Lessons Learned in Designing an Automated Data Science Pipeline using Short and Wide Obfuscated Navy Data"

✨ March 2024 -- Congratulations to the two recent alumni Md. Mouinul Islam and Mahsa Asadi for their accepted paper to SIGMOD 2024.

✨ Congratulations to Prof. Senjuti Basu Roy for the Outstanding Research Award for the academic year 2023 at the Ying Wu College of Computing.

✨ Our two papers, titled, "Cooperative Route Planning Framework for Multiple Distributed Assets in Maritime Applications", and "Rank Aggregation with Proportionate Fairness", have been awarded three prestigious badges from SIGMOD 2023 Artifacts & Reproducibility Committee: "Artifacts Available", "Artifacts Evaluated", and "Results Reproduced".

✨ Thank you SIGMOD Blog for hosting our article as part of their blog anniversary — Returning Top-K: Preference Aggregation or Sortition, or is there a Better Middle Ground?

✨ June 2022 -- Congratulations to Md. Mouinul Islam, Mahsa Asadi, and Dong Wei for their accepted papers to be presented at VLDB 2023.

✨ June 2022 -- Congratulations to PhD student [Dong Wei](Dong Wei), for defending his PhD thesis successfully, and winning the Joseph Leung Award for the Best CS PhD thesis of the year. Dong is in Google now!

✨ May 2022 -- Congratulations to the BDAL members - 2 accepted research papers at SIGMOD 2022, one research paper at ICDE 2022, and one accepted manuscript at VLDB Journal 2022.

✨ [Prof. Senjuti Basu Roy](https://web.njit.edu/~senjutib/index.html) is recognized as one of the 100 Early-Career Engineers Selected to Participate in NAE's 2021 US Frontiers of Engineering Symposium.

✨ March 2021 -- Matteo Lissandrini, Davide Mottin, Senjuti Basu Roy, and Yannis Velegrakis are co-organizing SEADATA 2021, co-located with VLDB 2021. Please consider submitting your work there.

✨ January 2021 -- SIGMOD 2021 Panel co-moderated with Sihem Amer-Yahia: Data Management to Social Science and Back in the Future of Work

✨ January 2021 -- Congratulations to Dong Wei for his ICDE 2021 paper titled "Peer Learning Through Targeted Dynamic Groups Formation"

✨ July 2020 -- Congratulations Dr. Esfandiari on completing PhD, wish you the very best for your future endeavours!

2025

  • Sepideh Nikookar, Sohrab Namazi Nia, Senjuti Basu Roy, Sihem Amer-Yahia, Behrooz Omidvar-Tehrani. "Model Reusability in Reinforcement Learning", to appear VLDB Journal 2025.
  • Gerald White, Deep Mistry, Kevin Chhoa, Senjuti Basu Roy, Lingyi Zhang, Adam Bienkowski, Krishna Pattipati. "A Computational Framework for Estimating Days of Maintenance Delay of Naval Ships", EDBT 2025. [PDF] [BibTex]

2024

  • Senjuti Basu Roy, Baruch Schieber, Nimrod Talmon. "Fairness in Preference Queries: Social Choice Theories Meet Data Management", VLDB 2024 (tutorial). [PDF] [BibTex]
  • Mouinul Islam, Mahsa Asadi, Senjuti Basu Roy. "Equitable Top-k Results for Long Tail Data", ACM SIGMOD 2024. [PDF] [BibTex]
  • Mouinul Islam, Soroush Vahidi, Baruch Schieber, Senjuti Basu Roy. "Promoting Fairness and Priority in k-Winners Selection Using IRV", ACM SIGKDD 2024. [PDF] [BibTex]
  • Vishal Chakraborty, Stacy Ann-Elvy, Sharad Mehrotra, Faisal Nawab, Mohammad Sadoghi, Shantanu Sharma, Nalini Venkatasubramanian, Farhan Saeed. "Data-CASE: Grounding Data Regulations for Compliant Data Processing Systems", Proceedings of the International Conference on Extending Database Technology (EDBT), 2024. [PDF] [BibTex]
  • Senjuti Basu Roy. "Fairness and Robustness in Answering Preference Queries", IEEE Data Engineering Bulletin 2024. [PDF] [BibTex]
  • Shanshan Han, Vishal Chakraborty, Michael T. Goodrich, Sharad Mehrotra, and Shantanu Sharma. "VEIL: A Storage and Communication Efficient Volume-Hiding Algorithm", ACM SIGMOD 2024. [PDF] [BibTex]
  • Shufan Zhang, Xi He, Ashish Kundu, Sharad Mehrotra, and Shantanu Sharma. "Secure Normal Form: Mediation Among Cross Cryptographic Leakages in Encrypted Databases", ICDE 2024. [PDF] [BibTex]
  • Shlomi Dolev, Komal Kumari, Sharad Mehrotra, Baruch Schieber, and Shantanu Sharma. "Brief Announcement: Make Master Private-Keys Secure by Keeping It Public", Proceedings of the International Symposium on Stabilization, Safety, and Security of Distributed Systems (SSS), 2024. [PDF] [BibTex]
  • Komal Kumari, Sharad Mehrotra, and Shantanu Sharma. "Tutorial: Information Leakage from Cryptographic Techniques", IEEE 44th International Conference on Distributed Computing Systems (IEEE ICDCS), 2024. [PDF] [BibTex]

2023

  • Mouinul Islam, Mahsa Asadi, Sihem Amer-Yahia, Senjuti Basu Roy. "A Generic Framework for Efficient Computation of Top-k Diverse Results", VLDB Journal 2023. [PDF] [BibTex]
  • Mouinul Islam, Dong Wei, Baruch Schieber, Senjuti Basu Roy. "Satisfying Complex Top-k Fairness Constraints by Preference Substitutions", VLDB 2023. [PDF] [BibTex]
  • Thinh On, Subhodeep Ghosh, Mengnan Du, Senjuti Basu Roy. "Proportionate Diversification of Top-k LLM Results using Database Queries", LLMDB Workshop at VLDB 2023. [PDF] [BibTex]
  • Shantanu Sharma, Yin Li, Sharad Mehrotra, Nisha Panwar, Peeyush Gupta, and Dhrubajyoti Ghosh. "Prism: Privacy-Preserving and Verifiable Set Computation over Multi-Owner Secret Shared Outsourced Databases", IEEE TDSC 2023. [PDF] [BibTex] [YouTube]
  • Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Shanshan Han, Shantanu Sharma, Chaoyang He, Sharad Mehrotra, and Salman Avestimehr. "Federated Analytics: A survey", APSIPA Transactions on Signal and Information Processing: Vol. 12: No. 1, e4, 2023. [PDF] [BibTex]
  • Shantanu Sharma, Yin Li, Sharad Mehrotra, Nisha Panwar, Komal Kumari, Swagnik Roychoudhury. "Information-Theoretically Secure and Highly Efficient Search and Row Retrieval", Proceedings of the VLDB Endowment (PVLDB), 2023. [PDF] [BibTex] [Code]
  • Dhrubajyoti Ghosh, Peeyush Gupta, Sharad Mehrotra, Shantanu Sharma. "Supporting Complex Query Time Enrichment For Analytics", Proceedings of the International Conference on Extending Database Technology (EDBT), 2023. [PDF] [BibTex]
  • Abhishek Singh, Yinan Zhou, Sharad Mehrotra, Mohammad Sadoghi, Shantanu Sharma, and Faisal Nawab. "WedgeBlock: An Off-Chain Secure Logging Platform for Blockchain Applications", Proceedings of the International Conference on Extending Database Technology (EDBT), 2023. [PDF] [BibTex]

2022

  • Sepideh Nikookar, Paras Sakharkar, Sathyanarayanan Somasunder, Senjuti Basu Roy, Adam Bienkowski, Matthew Macesker, David Sidoti, Krishna Pattipati. "Cooperative Route Planning Framework for Multiple Distributed Assets in Maritime Applications", SIGMOD 2022. [PDF] [BibTex]
  • Dong Wei, Mouinul Islam, Baruch Schieber, Senjuti Basu Roy. "Rank Aggregation with Proportionate Fairness", SIGMOD 2022. [PDF] [BibTex]
  • Sepideh Nikookar, Payam Esfandiari, Ria Mae Borromeo, Sihem Amer-Yahia, Senjuti Basu Roy. "Diversifying Recommendations on Sequences of Sets", VLDB Journal 2022. [PDF] [BibTex]
  • Sepideh Nikookar, Paras Sakharkar, Baljinder Smagh, Sihem Amer-Yahia, Senjuti Basu Roy. "Guided Task Planning Under Complex Constraints", ICDE 2022. [PDF] [BibTex]
  • Shantanu Sharma, Sharad Mehrotra, Nisha Panwar, Nalini Venkatasubramanian, Peeyush Gupta, Shanshan Han, and Guoxi Wang. "Quest: Privacy-Preserving Monitoring of Network Data: A System for Organizational Response to a Pandemic", IEEE Transactions on Services Computing, 2022. [PDF] [BibTex]
  • Dhrubajyoti Ghosh, Peeyush Gupta, Sharad Mehrotra, Shantanu Sharma. "A Case for Enrichment in Data Management Systems", SIGMOD Records, 2022. [PDF] [BibTex]
  • Peeyush Gupta, Sharad Mehrotra, Shantanu Sharma, Roberto Yus, and Nalini Venkatasubramanian. "Sentaur: Sensor Observable Data Model for Smart Spaces", Proceedings of the Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM), 2022. [PDF] [BibTex]

2021

  • Jees Augustine, Suraj Shetiya, Payam Esfandiari, Senjuti Basu Roy, Gautam Das. "A Generalized Approach for Reducing Expensive Distance Calls for A Broad Class of Proximity Problems", SIGMOD 2021. [PDF] [BibTex]
  • Dong Wei, Ioannis Koutis, Senjuti Basu Roy. "Peer Learning Through Targeted Dynamic Groups Formation", ICDE 2021. [PDF] [BibTex]
  • Mohammadreza Esfandiari, Ria Mae Borromeo, Sepideh Nikookar, Paras Sakharkar, Sihem Amer-Yahia, Senjuti Basu Roy. "Multi-Session Diversity to Improve User Satisfaction in Web Applications", TWC (formerly WWW) 2021. [PDF] [BibTex]
  • Nisha Panwar*, Shantanu Sharma*, Guoxi Wang, Sharad Mehrotra, Nalini Venkatasubramanian, Mamadou H. Diallo, and Ardalan Amiri Sani. "IoT Notary: Attestable Sensor Data Capture in IoT Environments", ACM Transactions on Internet Technology, 2021. [PDF] [BibTex]

2020

  • Dong Wei, Senjuti Basu Roy, Sihem Amer-Yahia. "Recommending Deployment Strategies for Collaborative Tasks", SIGMOD 2020. [PDF] [BibTex]
  • Dong Wei, Senjuti Basu Roy, Sihem Amer-Yahia. "Task Deployment Recommendation with Worker Availability" (poster), ICDE 2020. [PDF] [BibTex]
  • Sihem Amer-Yahia, Senjuti Basu Roy, Lei Chen, Atsuyuki Morishima, and others. "Making AI Machines Work for Humans in FoW", SIGMOD Record 2020. [PDF] [BibTex]
  • Idir Benouaret, Sihem Amer-Yahia, Senjuti Basu Roy, Christiane Kamdem Kengne, Jalil Chagraoui. "Enabling Decision Support Through Ranking and Summarization of Association Rules for TOTAL Customers", Trans. Large Scale Data Knowl. Centered Syst. 2020. [PDF] [BibTex]

2019

  • Senjuti Basu Roy. "Capturing Human Factors to Optimize Crowdsourced Label Acquisition through Active Learning", Special Issue of IEEE Data Engineering Bulletin, 2019. [PDF] [BibTex]
  • Sihem Amer-Yahia, Senjuti Basu Roy. "The Ever Evolving Online Labor Market: Overview, Challenges and Opportunities" (tutorial), VLDB 2019. [PDF] [Slides] [BibTex]
  • Mohammadreza Esfandiari, Dong Wei, Sihem Amer-Yahia, Senjuti Basu Roy. "Optimizing Peer Learning in Online Groups with Affinities", KDD 2019 (Acceptance rate 14.16%). [PDF] [BibTex]
  • Md. Abdus Salam, Mary Koone, S. Thirumuruganathan, Gautam Das, Senjuti Basu Roy. "A Human-in-the-loop Attribute Design Framework for Classification", WWW 2019 (Acceptance rate 18%). [PDF] [BibTex]

2018

  • Mohammadreza Esfandiari, Senjuti Basu Roy, Sihem Amer-Yahia. "Explicit Preference Elicitation for Task Completion Time", CIKM 2018. [PDF] [BibTex]
  • Habibur Rahman, Senjuti Basu Roy, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. "Optimized group formation for solving collaborative tasks", VLDB Journal 2018. [PDF] [BibTex]
  • Mohammadreza Esfandiari, Kavan Patel, Sihem Amer-Yahia, Senjuti Basu Roy. "Crowdsourcing Analytics with CrowdCur" (demo), SIGMOD 2018. [PDF] [BibTex]
  • Julien Pilourdault, Sihem Amer-Yahia, Senjuti Basu Roy, Dongwon Lee. "Task Relevance and Diversity as Worker Motivation in Crowdsourcing", ICDE 2018. [PDF] [BibTex]
  • Senjuti Basu Roy, Moushumi Maria, Tina Wang, Anne Ehlers, David Flum. "Predicting Adverse Events after Surgery through Sequence Modeling", Big Data Research, Special Issue 2018. [PDF] [BibTex]
  • Sihem Amer-Yahia, Senjuti Basu Roy. "Interactive Exploration of Composite Items" (tutorial), EDBT 2018. [PDF] [BibTex]

2017 

  • Julien Pilourdault, Sihem Amer-Yahia, Dongwon Lee, Senjuti Basu Roy. "Motivation-Aware Task Assignment in Crowdsourcing", EDBT 2017, 246-257
  • Habibur Rahman, Senjuti Basu Roy, Gautam Das. "A Probabilistic Framework for Estimating Pairwise Distances Through Crowdsourcing". EDBT 2017, 258-269
  • Sampoorna Biswas, Laks V. S. Lakshmanan, Senjuti Basu Roy. "Combating the Cold Start User Problem in Model-Based Collaborative Filtering". CoRR abs/1703.00397 (2017)

2016

  • Davide Mottin, Alice Marascu, Senjuti Basu Roy, Themis Palpanas, Yannis Velegrakis, Gautam Das. "A Holistic and Principled Approach for the Empty-Answer Problem", VLDB Journal 2016.
  • Kosetsu Ikeda, Atsuyuki Morishima, Habibur Rahman, Senjuti Basu Roy, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. "Collaborative Crowdsourcing with Crowd4U", VLDB 2016.
  • Sihem Amer-Yahia, Senjuti Basu Roy. "Human Factors in Crowdsourcing "(tutorial), VLDB 2016.
  • Senjuti Basu Roy, Tina Eliassi-Rad, Spiros Papadimitriou. "Fast Best-Effort Search on Graphs with Multiple Attributes", ICDE 2016.

​2015

  • ​Habibur Rahman, Senjuti Basu Roy, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. "Task Assignment Optimization in Collaborative Crowdsourcing", ICDM 2015.
  • Senjuti Basu Roy, Ankur Teredesai, Kiyana Zolfaghar, Rui Liu, David Hazel, Stacey Newman, Albert Marinez. "Dynamic Hierarchical Classification for Patient Risk-of-Readmission", ACM SIGKDD 2015 (industry and government track).
  • Habibur Rahman, Saravanan Thirumuruganathan, Senjuti Basu Roy, Sihem Amer-Yahia, Gautam Das. "Worker Skill Estimation in Team-Based Tasks", VLDB 2015.
  • Senjuti Basu Roy,Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, Gautam Das. "Task-Assignment Optimization in Knowledge Intensive Crowdsourcing", VLDB Journal 2015.
  • Senjuti Basu Roy, Laks V.S. Lakshmanan, Rui Liu. "From Group Recommendations to Group Formation", SIGMOD 2015.
  • Sihem Amer-Yahia, Senjuti Basu Roy. "From Complex Object Exploration to Complex Crowdsourcing", tutorial at WWW 2015.
  • Senjuti Basu Roy, Sihem Amer-Yahia, Lucas Joppa. "ECCO- A Framework for Ecological Data Collection and Management Involving Human Workers", EDBT 2015.
  • Sihem Amer-Yahia, Behrooz Omidvar-Tehrani, Senjuti Basu Roy, Nafiseh Shabib. "Group Recommendation with Temporal Affinities", EDBT 2015.

 

Director

Basu Roy, Senjuti

Basu Roy, Senjuti

Associate Professor

View Profile

Faculty Member

Sharma, Shantanu

Sharma, Shantanu

Assistant Professor

View Profile

Research Scientists

RS-Jerry
Gerald White
gkw@njit.edu
RS-Deep
Deep Ketan Mistry
dm728@njit.edu
RS-Kevin
Kevin Chhoa
kevin.chhoa@njit.edu

Research Students

phd-fatemeh
Fatemeh Ramezani
PhD Student (since 2024)
fr46@njit.edu
phd-sohrab
Sohrab Namazi Nia
PhD Student (since 2022)
sn773@njit.edu
phd-swastik
Swastik Biswas
PhD Student (since 2023)
sb2785@njit.edu
phd-thinh
Thinh On
PhD Student (since 2023)
to58@njit.edu
phd-subhodeep
Subhodeep Ghosh
PhD Student (since 2024)
sg2646@njit.edu

Alumni:

Will be updated soon.

1.S. Guo, A. G. Parameswaran, and H. Garcia-Molina. So who won?:dynamic max discovery with the crowd. In SIGMOD, pages 385–396,2012.

2.Data-driven crowdsourcing: Management, mining, and applications. In ICDE, pages 1527–1529. IEEE, 2015.

3. Das, P. S. G. C., A. Doan, J. F. Naughton, G. Krishnan, R. Deep,E. Arcaute, V. Raghavendra, and Y. Park. Falcon: Scaling up hands-off crowdsourced entity matching to build cloud services. In SIGMOD, pages 1431–1446, 20

4. S. B. Davidson, S. Khanna, T. Milo, and S. Roy. Using the crowd for top-k and group-by queries. In ICDT, pages 225–236, 2013.

5. A. Doan, M. J. Franklin, D. Kossmann, and T. Kraska. Crowdsourcing applications and platforms: A data management perspective.Proceedings of the VLDB Endowment, 4(12):1508–1509, 2011.

6. J. Fan, M. Zhang, S. Kok, M. Lu, and B. C. Ooi. Crowdop: Queryoptimization for declarative crowdsourcing systems.

IEEE Trans. Knowl.Data Eng., 27(8):2078–2092, 2015.

7. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin.Crowddb: answering queries with crowdsourcing. In SIGMOD, pages 61–72, 2011.

8. J. Gao, Q. Li, B. Zhao, W. Fan, and J. Han. Truth discovery and crowd sourcing aggregation: A unified perspective.Proceedings of theVLDB Endowment, 8(12):2048–2049, 2015.

9. S. Guo, A. G. Parameswaran, and H. Garcia-Molina. So who won?:dynamic max discovery with the crowd. InSIGMOD, pages 385–396,2012.

10. G. Li, C. Chai, J. Fan, X. Weng, J. Li, Y. Zheng, Y. Li, X. Yu, X. Zhang, and H. Yuan. CDB: optimizing queries with crowd-based selections and joins. In SIGMOD, pages 1463–1478, 2017.

11. G. Li, Y. Zheng, J. Fan, J. Wang, and R. Cheng. Crowdsourced data management: Overview and challenges. In SIGMOD, 2017.[16] A. Marcus, D. R. Karger, S. Madden, R. Miller, and S. Oh. Counting with the crowd.PVLDB, 6(2):109–120, 2012.

12. A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. Human-powered sorts and joins.PVLDB, 5(1):13–24, 2011.

13. A. Marcus, E. Wu, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, pages 211–214,2011.

14. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis,A. Ramesh, and J. Widom. Crowd Screen: algorithms for filtering data with humans. In SIGMOD, pages 361–372, 2012.

15. H. Park and J. Widom. Crowdfill: collecting structured data from the crowd. In SIGMOD, pages 577–588, 2014.

16. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. CrowdER: crowdsourcing entity resolution.PVLDB, 5(11):1483–1494, 2012.

17. J. Wang, G. Li, T. Kraska, M. J. Franklin, and J. Feng. Leveraging transitive relations for crowdsourced joins. InSIGMOD, 2013.

18. Kazemi, L., Shahabi, C., & Chen, L. (2013, November). Geotrucrowd: trustworthy query answering with spatial crowdsourcing. In Proceedings of the 21st acm sigspatial international conference on advances in geographic information systems (pp. 314-323). ACM.

19. To, H., Ghinita, G., & Shahabi, C. (2014). A framework for protecting worker location privacy in spatial crowdsourcing. Proceedings of the VLDB Endowment, 7(10), 919-930.

18. Roy, Senjuti Basu, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. "Crowds, not drones: modeling human factors in interactive crowdsourcing." 2013.

19. Rahman, H., Thirumuruganathan, S., Roy, S. B., Amer-Yahia, S., & Das, G. (2015). Worker skill estimation in team-based tasks. Proceedings of the VLDB Endowment, 8(11), 1142-1153.

20. Amer-Yahia, S., & Roy, S. B. (2016). Human factors in crowdsourcing. Proceedings of the VLDB Endowment, 9(13), 1615-1618.

21. Roy, S. B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2014). Optimization in knowledge-intensive crowdsourcing. arXiv preprint arXiv:1401.1302.

22. Ikeda, K., Morishima, A., Rahman, H., Roy, S. B., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2016). Collaborative crowdsourcing with crowd4U. Proceedings of the VLDB Endowment, 9(13), 1497-1500.

23. Morishima, A., Amer-Yahia, S., & Roy, S. B. (2014, September). Crowd4u: An initiative for constructing an open academic crowdsourcing network. In Second AAAI conference on human computation and crowdsourcing.

24. Esfandiari, M., Patel, K. B., Amer-Yahia, S., & Basu Roy, S. (2018, May). Crowdsourcing Analytics With CrowdCur. In Proceedings of the 2018 International Conference on Management of Data (pp. 1701-1704). ACM.

25. Esfandiari, M., Basu Roy, S., & Amer-Yahia, S. (2018, October). Explicit Preference Elicitation for Task Completion Time. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 1233-1242). ACM.

26. Esfandiari, M., Wei, D., Amer-Yahia, S., & Basu Roy, S. (2019, July). Optimizing Peer Learning in Online Groups with Affinities. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1216-1226). ACM.

27. Salam, M. A., Koone, M. E., Thirumuruganathan, S., Das, G., & Basu Roy, S. (2019, May). A Human-in-the-loop Attribute Design Framework for Classification. In The World Wide Web Conference (pp. 1612-1622). ACM.

28. Anagnostopoulos, A., Becchetti, L., Castillo, C., Gionis, A., & Leonardi, S. (2012, April). Online team formation in social networks. In Proceedings of the 21st international conference on World Wide Web (pp. 839-848). ACM.

28. Rahman, H., Roy, S. B., Thirumuruganathan, S., Amer-Yahia, S., & Das, G. (2019). Optimized group formation for solving collaborative tasks. The VLDB Journal—The International Journal on Very Large Data Bases, 28(1), 1-23.

29. Jacob Abernethy, Yiling Chen, and Jennifer Wortman Vaughan. Efficient market making via convex optimization, and a connection to online learning. ACM Transactions on Economics and Computation, 1(2):Article 12, 2013.

30. Ittai Abraham, Omar Alonso, Vasilis Kandylas, Rajesh Patel, Steven Shelford, and Alek- sandrs Slivkins. How many workers to ask? Adaptive exploration for collecting high quality labels. In SIGIR, 2016.

31. Arpit Agarwal, Debmalya Mandal, David C. Parkes, and Nisarg Shah. Peer prediction with heterogeneous users. In ACM EC, 2017.

32. Haas, D., Wang, J., Wu, E., & Franklin, M. J. (2015). Clamshell: Speeding up crowds for low-latency data labeling. Proceedings of the VLDB Endowment, 9(4), 372-383.

33. Omar Alonso. Implementing crowdsourcing-based relevance experimentation: An industrial perspective. Information Retrieval, 16(2):101–120, 2013.

34.. Omar Alonso, Daniel E. Rose, and Benjamin Stewart. Crowdsourcing for relevance evalu- ation. ACM SigIR Forum, 42(2):9–15, 2008.

35. Vamshi Ambati, Stephan Vogel, and Jaime Carbonell. Collaborative workflow for crowd- sourcing translation. In CSCW, 2012.

36. Paul Andr´e, Haoqi Zhang, Juho Kim, Lydia B. Chilton, Steven P. Dow, and Robert C. Miller. Community clustering: Leveraging an academic crowd to form coherent conference sessions. In HCOMP, 2013

37. Julia Angwin, Je Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There’s software used across the country to predict future criminals and it’s biased against blacks. ProPublica article accessed at https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing, 2016

38. Pavel D. Atanasov, Phillip Rescober, Eric Stone, Samuel A. Swift, Emile Servan-Schreiber, Philip E. Tetlock, Lyle Ungar, and Barbara Mellers. Distilling the wisdom of crowds: Prediction markets versus prediction polls. Management Science, 63(3):691–706, 2017.

39. Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, and Murat Demir- bas. Crowdsourcing for multiple-choice question answering. In AAAI, 2014.

40. Solon Barocas and Andrew Selbst. Big data’s disparate impact. California Law Review, 104, 2016.

Jonathan Baron, Barbara A. Mellers, Philip E. Tetlock, Eric Stone, and Lyle H. Ungar. Two reasons to make aggregated probability forecasts more extreme. Decision Analysis, 11(2):133–145, 2014.

41. Joyce Berg, Robert Forsythe, Forrest Nelson, and Thomas Rietz. Results from a dozen years of election futures markets research. Handbook of experimental economics results, 1:742–751, 2008.

42. Michael Bernstein, Greg Little, Rob Miller, Bjoern Hartmann, Mark Ackerman, David Karger, David Crowell, and Katrina Panovich. Soylent: A word processor with a crowd inside. In UIST, 2010.

43. Anant Bhardwaj, Juho Kim, Steven P. Dow, David Karger, Sam Madden, Robert C. Miller, and Haoqi Zhang. Attendee-sourcing: Exploring the design space of community-informed conference scheduling. In HCOMP, 2014.

44. Jeffrey P. Bigham. Reaching dubious parity with hamstrung hu- mans. Blog post accessed at http://jeffreybigham.com/blog/2017/ reaching-dubious-parity-with-hamstrung-humans.html, 2017.

45. David M. Blei and John D. Lafferty. Topic models. Text mining: Classification, clustering, and applications, 10(71):34, 2009.

46. David M. Blei, Andrew Y. Ng, , and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993–1022, 2003.

47. Jordan Boyd-Graber, Yuening Hu, and David Mimno. Applications of topic models. Foun- dations and Trends in Information Retrieval, 11(2–3):143–296, 2017.

48. Jonathan Bragg, Mausam, and Daniel S. Weld. Crowdsourcing multi-label classification for taxonomy creation. In HCOMP, 2013.

49. Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, and Patrice Simard. FeatureInsight: Visual support for error-driven feature ideation in text classification. In IEEE VAST, 2015.

50. Michael Buhrmester, Tracy Kwang, and Samuel D. Gosling. Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1):3–5, 2011.

51. Chris Callison-Burch and Mark Dredze. Creating speech and language data with Amazon’s Mechanical Turk. In NAACL HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, 2010.

52. Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, and Kate Crawford. AI Now 2017 Report. Accessed at https://ainowinstitute.org/AI_Now_2017_Report.pdf, 2017.

53. Logan Casey, Jesse Chandler, Adam Seth Levine, Andrew Proctor, and Dara Z. Strolovitch. Intertemporal differences among MTurk worker demographics. Working paper on PsyArXiv, 2017.

54. Dana Chandler and Adam Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior and Organization, 90:123–133, 2013.

55. Jesse Chandler, Pam Mueller, and Gabriele Paolacci. Nonna¨ıvet´e among Amazon Mechan- ical Turk workers: Consequences and solutions for behavioral researchers. Behavior Re- search Methods, 46(1):112–130, 2014.

56.Jesse J. Chandler and Gabriele Paolacci. Lie for a dime: When most prescreening responses are honest but most study participants are imposters. Social Psychological and Person- ality Science, 8(5):500–508, 2017.

57.Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. Reading tea leaves: How humans interpret topic models. In NIPS, 2009.

58. Shuchi Chawla, Jason D. Hartline, and Balasubramanian Sivan. Optimal crowdsourcing contests. Games and Economic Behavior, 2015.

59. Yiling Chen and David M. Pennock. A utility framework for bounded-loss market makers. In UAI, 2007.

60. Yiling Chen and Jennifer Wortman Vaughan. A new understanding of prediction markets via no-regret learning. In ACM EC, 2010.

61. Yiling Chen, Arpita Ghosh, Michael Kearns, Tim Roughgarden, and Jennifer Wortman Vaughan. Mathematical foundations of social computing. Communications of the ACM, 59(12):102–108, December 2016.

62. Lydia Chilton, Juho Kim, Paul Andr´e, Felicia Cordeiro, James Landay, Dan Weld, Steven P. Dow, Robert C. Miller, and Haoqi Zhang. Frenzy: Collaborative data organization for creating conference sessions. In CHI, 2014.

63. Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. Cascade: Crowdsourcing taxonomy creation. In CHI, 2013.

64. Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidi- vism prediction instruments. Big Data, Special Issue on Social and Technical Trade-Offs, 2017.

65. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In KDD, 2017.

66. Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous proficiency. In WWW, 2013.

67. Susan B. Davidson, Sanjeev Khanna, Tova Milo, and Sudeepa Roy. Top-k and clustering with noisy comparisons. ACM Transactions on Database Systems, 39(4):35:1–39, 2014.

68. Philip Dawid and Allan Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C (Applied Statistics), 28(1):20–28, 1979.

69. Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudr´e-Mauroux. Zencrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW, 2012.

70. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In CVPR, 2009.

71. Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1):114, 2015.

72. Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. Overcoming algorithm aver- sion: People will use imperfect algorithms if they can (even slightly) modify them. Man- agement Science, 2016.

73. Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G. Ipeirotis, and Philippe Cudr´e-Mauroux. The dynamics of micro-task crowdsourcing: The case of Ama- zon MTurk. In WWW, 2015.

74. Dominic DiPalantino and Milan Vojnovic. Crowdsourcing and all-pay auctions. In ACM EC, 2009.

75. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. CoRR arXiv:1702.08608, 2017.

76. Mary T. Dzindolet, Linda G. Pierce, Hall P. Beck, and Lloyd A. Dawe. The perceived utility of human and automated aids in a visual detection task. Human Factors, 44(1): 79–94, 2002.

77. Robert C. Edgar and Serafim Batzoglou. Multiple sequence alignment. Current opinion in structural biology, 16(3):368–373, 2006.

78.Ju Fan, Guoliang Li, Beng Chin Ooi, Kian-lee Tan, and Jianhua Feng. icrowd: An adaptive crowdsourcing framework. In SIGMOD, 2015

79. Oluwaseyi Feyisetan, Elena Simperl, Max Van Kleek, and Nigel Shadbolt. Improving paid microtasks through gamification and adaptive furtherance incentives. In WWW, 2015.

80. Urs Fischbacher and Franziska F¨ollmi-Heusi. Lies in disguise: An experimental study on cheating. Journal of the European Economic Association, 11(3):525–547, 2013.

81. Chao Gao, Yu Lu, and Dengyong Zhou. Exact exponent in optimal rates for crowdsourcing.In ICML, 2016.

82. Xi Alice Gao, Yoram Bachrach, Peter Key, and Thore Graepel. Quality expectation-variance tradeoffs in crowdsourcing contests. In AAAI, 2012.

83. Xi Alice Gao, Andrew Mao, Yiling Chen, and Ryan Prescott Adams. Trick or treat: Putting peer prediction to the test. In ACM EC, 2014.

84. Yashesh Gaur, Florian Metze, Yajie Miao, and Jeffrey P. Bigham. Using keyword spotting to help humans correct captioning faster. In INTERSPEECH, 2015.

85. Yashesh Gaur, Florian Metze, and Jeffrey P. Bigham. Manipulating word lattices to incor- porate human corrections. In INTERSPEECH, 2016.

86. Timnit Gebru, Jonathan Krause, Jia Deng, and Li Fei-Fei. Scalable annotation of fine- grained objects without experts. In CHI, 2017.

87. Arpita Ghosh, Satyen Kale, and Preston McAfee. Who moderates the moderators? Crowd- sourcing abuse detection in user-generated content. In ACM EC, 2011.

87. Daniel G. Goldstein, R. Preston McAfee, and Siddharth Suri. The cost of annoying ads. In WWW, 2013.

88. Daniel G. Goldstein, Siddharth Suri, R. Preston McAfee, Matthew Ekstrand-Abueg, and Fernando Diaz. The economic and cognitive costs of annoying display advertisements. Journal of Marketing Research, 51(6):742––752, 2014.

89. Ryan Gomes, Peter Welinder, Andreas Krause, and Pietro Perona. Crowdclustering. In NIPS, 2011.

90. Joseph K. Goodman and Gabriele Paolacci. Crowdsourcing consumer research. Journal of Consumer Research, 44(1):196–210, 2017.

91. Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. The crowd is a collaborative network. In CSCW, 2016.

92. Neha Gupta, David Martin, Benjamin V. Hanrahan, and Jacki O’Neil. Turk-life in India. In the Intnernational Conference on Supporting Groupwork, 2014.

93. Juho Hamari, Jonna Koivisto, and Harri Sarsa. Does gamification work? – A literature review of empirical studies on gamification. In Hawaii International Conference on System Sciences, 2014.

94. Robin Hanson. Combinatorial information market design. Information Systems Frontiers, 5(1):105–119, 2003.

95. Christopher G. Harris. You’re hired! An examination of crowdsourcing incentive models in human resource tasks. In WSDM Workshop on Crwdsourcing for Search and Data Mining, 2011.

96. Jeffrey Heer and Michael Bostock. Crowdsourcing graphical perception: Using Mechanical Turk to assess visualization design. In CHI, 2010.

Hannes Heikinheimo and Antti Ukkonen. The crowd-median algorithm. In HCOMP, 2013.

97. Chien-Ju Ho and Jennifer Wortman Vaughan. Online task assignment in crowdsourcing markets. In AAAI, 2012.

98. Chien-Ju Ho, Shahin Jabbari, and Jennifer Wortman Vaughan. Adaptive task assignment for crowdsourced classification. In ICML, 2013.

99. Chien-Ju Ho, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. Incen- tivizing high quality crowdwork. In WWW, 2015.

100. Chien-Ju Ho, Aleksandrs Slivkins, and Jennifer Wortman Vaughan. Adaptive contract de- sign for crowdsourcing markets: Bandit algorithms for repeated principal-agent problems. Journal of Artificial Intelligence Research, 55:317–359, 2016.

101. John J. Horton, David Rand, and Richard Zeckhauser. The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3):399–425, 2011.

102. Yuening Hu, Jordan Boyd-Graber, Brianna Satinoff, and Alison Smith. Interactive topic modeling. Machine Learning, 95:423–469, 2014.

103. Lilly C. Irani and M. Six Silberman. Turkopticon: Interrupting worker invisibility in Ama- zon Mechanical Turk. In CHI, 2013.

104. Ayush Jain, Akash Das Sarma, Aditya Parameswaran, and Jennifer Widom. Understanding workers, developing effective tasks, and enhancing marketplace dynamics: A study of a large crowdsourcing marketplace. Proceedings of the VLDB Endowment, 10(7):829–840, 2017.

105. Jongbin Jung, Connor Concannon, Ravi Shro, Sharad Goel, and Daniel G. Goldstein. Sim- ple rules for complex decisions. CoRR arXiv:1702.04690, 2017.

106. Radu Jurca and Boi Faltings. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research, 34:209–253, 2009.

107. Ece Kamar. Directions in hybrid intelligence: Complementing AI systems with human intelligence. Abstract for IJCAI Early Career Spotlight Track Talk, 2016.

108. Ece Kamar and Eric Horvitz. Incentives for truthful reporting in crowdsourcing (short paper). In AAMAS, 2012.

Subscribe to