[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » pml-haters

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
pml-haters [2007/05/16 03:19]
bojar odkaz na Honzovu ukazku validace 		pml-haters [2007/06/05 06:35] (current)
bojar jen formatovani
Line 6: Line 6:
  
 Links to additional tools are at the bottom of the page. Links to additional tools are at the bottom of the page.
 +
 +I strongly recommend [[http://www.jucs.org/jucs_3_9/why_we_need_an/Prechelt_L.pdf|Lutz Prechelt: Why We Need an Explicit Forum for Negative Results]] (esp. section 2.6) for an explanation why not speaking about weaknesses hurts.
  
 ===== In Spite of some Common Assumptions... ===== ===== In Spite of some Common Assumptions... =====
Line 12: Line 14:
  
 Please strongly prefer SAX-based tools to DOM-based tools. Please strongly prefer SAX-based tools to DOM-based tools.
- 
  
  
 ===== Validation ===== ===== Validation =====
  
-Given a PML file, how do I validate it? I always forget... Please provide me with the one-liner to do the validation.+Given a PML file, how do I validate it?
  
-See [[user:ptacek:tectomt|this snippet]] for some vague hints.+For most purposes, a libxml2 (DOM) based validator 
 +<code>/f/common/exec/validate_pml --pml-dir /f/common/share/pml --path /f/common/share/tred file_to_validate</code>should work fine and fast. 
 + 
 +For huge files, use<code>/f/common/exec/validate_pml_stream --path /f/common/share/tred file_to_validate</code>which is based on Jing (SAX); Jing has no Zlib or stdin support, so some space in /tmp will be needed for temporary files. 
 +Both scripts have decent user documentation. See inside the scripts if interested in the implementation details.
  
 ===== XSH Won't Work: Blame XML Namespaces ===== ===== XSH Won't Work: Blame XML Namespaces =====
Line 40: Line 45:
 The reason why ''cd tdata'' won't work is a badly specified XML namespace. This works: The reason why ''cd tdata'' won't work is a badly specified XML namespace. This works:
  
-<code>$f/>regns pml http://ufal.mff.cuni.cz/pdt/pml/+<code>$f/>regns pml http://ufal.mff.cuni.cz/pdt/pml/;
 $f/>cd pml:tdata $f/>cd pml:tdata
 $f/pml:tdata> $f/pml:tdata>
 </code> </code>
 +
 +Hint: add the regns command to your ~/.xsh2rc.
  
 You will have to write the ''pml:'' prefix before every tag name in every XPath! You will have to write the ''pml:'' prefix before every tag name in every XPath!
  
 Most probably you'll still face problems when accessing attributes of XML elements, because namespacing rules apply differently to attributes and elements. You'll need to read XML (Namespaces) specification. Most probably you'll still face problems when accessing attributes of XML elements, because namespacing rules apply differently to attributes and elements. You'll need to read XML (Namespaces) specification.
- 
- 
- 
- 
- 
- 
- 
  
 ===== Number of Sentences ===== ===== Number of Sentences =====
Line 64: Line 64:
 This XPath would quickly give you the number of sentences: This XPath would quickly give you the number of sentences:
  
-<code>count(tdata/trees/LM)+<code>count(pml:tdata/pml:trees/pml:LM)
 </code> </code>
  
Line 73: Line 73:
 <code>cat file.t.xml \ <code>cat file.t.xml \
 | xsh -I - -C "regns pml http://ufal.mff.cuni.cz/pdt/pml/; count(pml:tdata/pml:trees/pml:LM)" 2>/dev/null | xsh -I - -C "regns pml http://ufal.mff.cuni.cz/pdt/pml/; count(pml:tdata/pml:trees/pml:LM)" 2>/dev/null
 +</code>
 +
 +or just the following, if you have the regns command in your ~/.xsh2rc:
 +
 +<code>cat file.t.xml \
 +| xsh -I - -C "count(pml:tdata/pml:trees/pml:LM)" 2>/dev/null
 </code> </code>
  
Line 88: Line 94:
 </code> </code>
  
 +Here is a one-liner in Perl that does not load the whole file into memory:
  
 +<code>perl -MXML::LibXML::Reader -e 'my $r=XML::LibXML::Reader->new(location=>shift); $r->nextElement("trees"); $d=$r->depth; $r->read; while ($d<$r->depth) {$i++; $r->nextSibling} print $i,"\n"' file.t.xml 
 +</code>
  
 ===== Restricting a Suite of PML Files to Contain only a Specific Sentence ===== ===== Restricting a Suite of PML Files to Contain only a Specific Sentence =====
Line 97: Line 105:
 How do I create a suite of files with just the problematic sentence 345, i.e. files test-w.xml, test-m.xml, test-a.xml and test-t.xml, all properly referenced? A XML-Reader based script by Petr Pajas demonstrates that: How do I create a suite of files with just the problematic sentence 345, i.e. files test-w.xml, test-m.xml, test-a.xml and test-t.xml, all properly referenced? A XML-Reader based script by Petr Pajas demonstrates that:
  
-<code>~pajas/projects/pml/separate_t_tree.pl file-t.xml 345+<code>~pajas/projects/pml/tools/separate_t_tree.pl file-t.xml 345
 </code> </code>
  
Line 107: Line 115:
  
 [[http://www.ltg.ed.ac.uk/software/xml/|LT XML]] - tools like sggrep, sgcount, knit... for handling SGML files on command-line, **SAX-based**, i.e. can handle big files [[http://www.ltg.ed.ac.uk/software/xml/|LT XML]] - tools like sggrep, sgcount, knit... for handling SGML files on command-line, **SAX-based**, i.e. can handle big files
- 

[ Back to the navigation ] [ Back to the content ]