CSCI 736 Research Final

CSCI 736 NN&ML (Neural Computation): Research Project

In statistical learning research, after months (or years) of coding and experimentation with a novel idea or application, it is common to publish one's idea and experimental findings. This project aims to simulate that process over the course of the semester, yielding the kernel that you could potentially take a few steps further to publish at a conferences such as the Conference on Neural Information Processing Systems (NIPS, or nowadays renamed to NeurIPS) or journals such as the Journal of Machine Learning Research (JMLR).

For this work, you will work in teams of 1-3 students (those you have been assigned for paper presentations). The goal of this assignment is for your group to conduct research in the area of artificial neural systems. This project can be approached in one of three ways:

Reproducibility Study: You can select a paper (or related set of papers), strongly related to the topics covered in this class, and implement the models/algorithms yourselves and reproduce the results of the target paper(s). Note that you are also required to try out your implementation on another (perhaps related) dataset that the paper did not use itself.
Application Study: You can apply several of the various algorithms or models covered in this course to an interesting problem of your choice, meanning that you will need obtain a dataset for your problem. Note this will require careful specification of the problem/task.
Fundamental Work: You can design your own neural model and/or learning algorithm (or variant/hybrid of those we have studied) and evaluate it on relevant benchmark datasets. Note that you will have to evaluate your idea rigorously to illustrate some form of generality or apply it to an actual dataset/problem to show its value on a specific problem type. You will need to compare your idea to at least a few several key/relevant baselines (unless you can find prior results reported on an actual dataset you can use), which will depend on your problem/application.

Note: Your project could be some combination of the above types, but generally it is good to propose with a focus and then adapt as your project work evolves throughout the semester.

Literature & Topic Selection

Your papers should come from reputable sources (conferences or journals) related to the topic at hand. Your topic should be related to the course content. You can use the library database to find papers - look in the ACM Digital Library or IEEE Explore databases for best results. Another place to consider is in the DL textbook, which contains an extensive references/bibliography. For those papers, you might also use Google Scholar or CiteSeerX to find forward citations (i.e. those papers that cite the mentioned paper) which will be newer and may be more interesting. Depending on which type of project you choose to do, the papers you point me to will be different. For a reproduciblity study, this is simple -- I expect the paper(s) you will be implementing/reproducing. For an application study, I expect at least 1-2 papers that describe the problem you want to approach and/or related work. For a fundamental work project, I expect at least 2 papers that describe work related/relevant to your idea(s).
Note: Models/algorithms developed or problem(s) being solved should be of sufficient complexity/interest fit for a semester-long team project.

Proposal

Write-up: You will write and submit to MyCourses a brief, concise document (PDF, 2 pages of text content, more if including figures/visuals) describing/answer the following:

the type of project you will conduct (see above; note that hybrids/combinations are permitted)
motivation - why should we care? why is this worth pursuing or interesting to the class? (relevance to topic/course and motivation from a scientific point-of-view)
a clear problem description (mix of relevant mathematics and clear writing)
relevant papers (citations with links or attached PDFs of relevant pubs), with brief description of the key ones
a rough timeline / basic plan of what you will do with the time alloted (Suggestion: a visual/figure goes well with this)
an overview / description of the overall study to be conducted, e.g., potential experimental design, data to be used, etc.

Presentation: You will be required to deliver an 15-20 minute (depends on how many teams per day end up presenting) presentation describing what your research project will be. You may use Powerpoint-style slides or not (alternatives include: Beamer, Keynote), as you see best to present your proposed work. I will expect to at least see the same points mentioned above for the write-up in the proposal presentation as well. The presentation slides are due 20 mintues before February 28 class.

Peer Evaluation: You will be required to provide feedback to your fellow classmates. Your feedback for the proposal should include suggestions/comments on the proposed work and at least 2 questions for the presenters to consider going forward in the semester with their work. Feedback must be typed (could be put in a *.txt or clean PDF, this is up to you). Your peer feedback will be compiled into one document so you can adjust/integrate what your classmates provide to you going forward in the semester. Peer review is due 11:59am, March 1 & March 3 (the noon of the day after each proposal presentation day). Please submit your peer review comments/feedback (and questions) on MyCourses (link to be shared during class).

Deliverable(s): The proposal must submitted to a MyCourses dropbox by 11:59pm February 27. Feedback must be submitted to the MyCourses dropbox by noon of the following day of the class that the presentation was given. Slides must be submitted to dropbox no later than 20 minutes before the class of the day that the presentation is to be given.
You will be expected to use LaTex to write your proposal (which will help you start the final paper) and to use NIPS format. The basic template and style files have been provided to you on this project page for you to readily use on something like Overleaf (which you can create a free account on). Note that you do not have to use Overleaf if you have a different tool/site you prefer to use for writing/compiling LaTex, e.g., MiKTeX.

Final Project

Project Materials/Code: You will need to write code for whatever you need to implement in Python. I will expect you to become familiar or already are familiar with version control software, particularly creating yourself a free GitLab or Github account (make sure you have the ability to create a private repo). You will be required to write a basic README describing how one should use your code to execute and reproduce your experimental results. This might mean you will want to write some basic Bash shell scripts to wrap up functionality if this will make it easier to use your code. We will need to be able to execute your code and it will be expected to compile/run correctly. So be sure to write good, clean code that you or others could use to experimentally explore your work/study further and specifically be able to reproduce the results you obtiained if they had the (computing) resources you used.

Deliverable: A zip file containing the README and all necessary, organized source code to be submitted to MyCourses by May 9, 11:59pm.

Final Presentation & Writeup: You need to make sure your talk/presentation adheres to the Heilmeier format for (technical) research proposals, answering as many of the questions found here as relevant. Note that not every single question will be relevant to your paper/talk, so adapt it to your needs as necessary. Broadly, I will be checking for: 1) background/clear problem description, 2) prior/related work/methods, 3) approach/project design/steps, 4) experimental design, 5) experimental results/comparisons, 6) insights/observations/lessons learned, 7) criticisms/future work/next steps.
Your slides can be submitted in PPT or PDF form (if you're using Beamer to create your slides, just make sure you compile them to PDF before final submission to MyCourses later). Slides are due 20 minutes before April 22 first presentation slot (1:10pm) with a chance for submitting an edited version by May 9 11:59pm.

Deliverable: A (compiled) PDF copy of your (typeset) final report and your presentation slides. These will be submitted to a MyCourses dropbox. You will be expected to use LaTex to write your final report and to use NIPS format. You will very likely want to just expand your proposal document (since you will have written the proposal with the right template provided above). Your paper should make sure that it answers the questions that your presentation/talk also tries to answer (Heilmeier format). However, its general structure should approximately follow that of what you would submit to a conference for publication. In essence, this means your writeup should have sections with content organized as follows:

Introduction -- contains motivation, background
Related work -- literature review, describes related work or prior work
Problem description -- formal description/presentation of the problem to be solved, data
Proposed solution -- present the method(s) you propose or reproduce clearly/formally, might describe baselines here as relevant
Experimental results/design -- benchmarks/baseline methods you will compare to, experiments you will conduct (things you will test for), data you will use
Discussion -- includes insights, lessons learned, observations based on experiments conducted in prior section
Conclusion -- should include future work

You will hand in any slides that you use to a MyCourses dropbox. The presentation will be graded on content (i.e. appropriate level for your fellow students), materials as appropriate, and presentation style (coherence, clarity, ability to answer questions). The final report document is due May 9, 11:59pm

Peer Evaluation: Your feedback for the final project should include suggestions/comments on the presented final work (including presentation). Feedback must be typed (could be put in a *.txt or clean PDF, this is up to you). All peer feedback for a particular presentation will be compiled into one document and provided to the team at the end of semester to faciliate final reflections and considerations that should be taken if the team wants to carry the paper forward to a machine learning journal.

Deliverable: Feedback (could be put in a *.txt or clean PDF, this is up to you) submitted to MyCourses the day of the relevant presentation at end of class.

Team Evaluation: you will be required to submit a bit of text outlining your team contributions and evaluating your team performance. In this text, you must answer 2 questions: 1) what did you specifically focus on / contribute to the project (code-wise, paper-wise, etc.), and 2) how would you describe and evaluate your team performance qualitatively (in a few sentences)? This is where you can list any possible issues or concerns and/or strengths/positives.

Deliverable: A block of text (*.txt or PDF) submitted to MyCourses containing the relevant feedback/questions/suggestions. The evaluation is due May 9, 11:59pm.

Grading Breakdown:

Proposal presentation: 15%
Executable code base: 25%
Final report: 35%
Final Presentation: 20%
Peer Evaluation: 5% (3% for proposal feedback, 2% for final project feedback)

Additional Notes On Peer & Team Evaluation

Peer Evaluation: To do well in machine learning research, you must learn to cope and deal with the process of peer review, often in the form of review by anonymous reviewers that decide if your work is to be accepted to or rejected from a conference/journal. Since you are to convey clearly to your classmates the content of your project/work, you will be receiving constructive feedback from your classmates on both your proposal and final project presentations. This feedback should be brief/concise and include suggestions for / commentary on the project content and presentation clarity.
Crucially, part of your project grade will include providing feedback to your fellow classmates (each student is expected to provided N-1 brief feedback text blocks, where N is the number of total student teams).