File Formats

• .sen format, for sentences: one sentence per line, separating all tokens by whitespace. Be sure to put spaces before and after all punctuation symbols!
• .gr format, for grammar rules: one rule per line, in the format:
  weight X Y1 Y2 ... Yn
  signifying the rule X -> Y1 Y2 ... Yn (where X is a non-terminal and the Ys are any mix of terminals and non-terminals). Anything on a line after a # is considered a comment.

Tools

• Random sentence generator:
  randsent [ -t ] [ -n num-sentences ] [ -s start-symbol ] grammar-files
  
  • -t: produce tree output instead of flat sentences.
  • -n num-sentences: e.g., -n 10 will generate ten sentences instead of just one.
  • -s start-symbol: The generator's start symbol is S1 by default. You may want to specify another so you can test just part of the grammar.
  • grammar-files: Typically *.gr. In general, a list of .gr files containing the parts of the grammar. Different parts may be written by different people.
• Parser:
  parse [-s start-symbol] sentence-file grammar-files
  
  • -s start-symbol: The parser's default start symbol is START. You may want to specify another.
  • sentence-file: Name of a file with one input sentence per line. The special filename - means to read the input sentences from the keyboard instead (or in general, from standard input).
  • grammar-files: As above.

  If a sentence fails to parse, its output will be NONE on a single line. If the parse is successful, you will see the single best-scoring (highest-weight) parse on one line, followed by the negative log-probability of the sentence according to your model. You will also see warnings, such as words in the sentences that are missing in your grammar.

  If you want to make the parses a bit more readable, pipe the output of parse to prettyprint.
  parse sentence-file *.gr | prettyprint

  If you want to see how you're doing in terms of cross-entropy, pipe the output of parse to crossent:
  parse sentence-file *.gr | crossent

Getting Started

1. Log in to one of the workshop machines; these should be running SunOS 5.7 or 5.8.
2. Add the bin directory to your path so that you can access the tools:
   If you use tcsh: setenv PATH /export/ws03_mt_2/scratch/ws03lab/bin:$PATH
If you use bash: export PATH=/export/ws03_mt_2/scratch/ws03lab/bin:$PATH
Please use tcsh or bash for this exercise, not csh. We're pretty sure that the default on your new account is tcsh.
3. To make it so that other people on your team can edit files you create, run
   umask 000
   
4. Wave your hands and flag down Noah to give your team a secretly-named directory if he hasn't already. This is where you should do all your work on this lab. Go to your team directory under /export/ws03_mt_2/scratch/ws03lab/. Someone should copy the files you need to get started:
   cp ../start/* ./
The script test.csh will do this for you.
5. Create a file called members that contains the (complete) email addresses of everyone on your team, one per line. This is so that we can give you sentence for grammaticality judgements later in the lab.
6. Use an editor such as Emacs to edit and create grammar (.gr) files. A good place to start is by modifying Top.gr, where the weights for S1 and S2, and the initial S1, are. We recommend that you not change the file Vocab.gr, but rather create new Tag -> word rules in addition to the ones we've given you, in a different file. You will want to use parse to see how well you can parse training data from the other teams, and you will want to use randsent to see if the sentences you are generating look like English.
7. When time is up, you will be asked to give grammaticality judgements on some sentences generated by various teams' grammars (you won't know where each sentence came from -- it could be yours -- so be honest!). When we quit, be sure you cat together all your grammar files into one file called GRAMMAR.gr so we can evaluate your model. Don't forget to include files like S2.gr and Vocab.gr when you do this! Then, to make sure everything will run in the evaluation, run:
   randsent -n 10 GRAMMAR.gr
parse examples.sen GRAMMAR.gr
8. We will spend the final part of the lab in discussion while the cross-entropy scores are computed.

Tips

• Familiarize yourself with the vocabulary. Think about useful ways to re-organize it so that words that have similar grammatical behavior are together -- consider changing the tag set for S1 (Noun, Verb, Misc, etc.) to something more complete and realistic.
• Divide up the work! Some tasks that you might be able to do in parallel:
  • Find tough sentences and sketch out trees for them (on paper)
  • Monitor team progress on the examples and the data from the other teams, generating a list of fragments you're not getting
  • Generate sentences and figure out what in the grammar is letting gibberish get through.
  • Think about ways to make S2 (that's right, the safety net grammar) smarter; perhaps there is a way to modify the weights so that while every sentence is possible, better sentences get more probability (even under S2!).
• Let each person have his/her own grammar file, so nobody over-writes anyone else's work. You'll need to agree on some conventions about non-terminals; for example, if you add a non-terminal, incorporate your initials into it so it doesn't accidentally get confused with others in the grammar.
• Some fairly autonomous regions of the grammar are noun phrases, questions, verb argument structure, and subject-verb agreement.
• The weights are important! We won't be generating very many sentences for evaluation.
• The biggest decision your team makes will be how to weight START -> S1 and START -> S2. How much do you trust your grammar?

Important Rules

• Preserve the non-terminals START, S1, and S2 for evaluation purposes - don't use them elsewhere.
• New vocabulary is strictly forbidden! If you add new vocabulary, the parser and sentence generator will warn you about it.

Warnings

• Be careful not to name any of your non-terminals the same as any terminals (this happened to people last year).
• Make sure all of your intended non-terminals have expansion rules; otherwise the parser will think they are terminals and they'll show up as new vocabulary.