GeSHi Bug Tracker - GeSHi
Viewing Issue Advanced Details
84 core minor always 05-26-06 20:04 05-28-06 21:16
nigel  
nigel  
low  
assigned 1.1.1alpha4  
open  
none    
none  
0000084: Provide access to tokens through an iterator
Florian wants to be able to have access to the tokens produced so he can perform actions like looking for functions in the source. In general I think this is a good idea, but memory limits must be taken into account.

It should be possible to iterate through the tokens as produced. However the person may or may not want the highlighted source also, and this should be taken into account.
child of 0000085assigned nigel Preview and Postview Functions for Code Parsers 

Notes
(0000412)
BenBE   
05-28-06 00:24   
SO you mean when I get to highlight something I get a complete list of all tokens of that source level (subcontexts are not splittet) and I can iterate through them??? i.e. I get a list of function \ space \ ident \ bracket ident ... and so on???

I think this could be combined with the context preview \ postview functions I suggested once ... Thus if you require this information you can request it within the preview method that is called before each subcontext is split for the CP.

Thus behaviour would be:
- Do splitting the context
- Call the CP preview for the parent context ("global scope" if this ist the top-most one)
- If the given context is to be split again repeat this behaviour for it recursively)
- Else call the CP for each of the subcontexts
- Call the CP postview function for the parent context (Does some clean-u)
(0000413)
nigel   
05-28-06 21:16   
Not quite :). What I mean is something like this:

$geshi =& new GeSHi($source, $language);
while ($token = $geshi->nextToken()) {
  // do what you want with $token
}

The idea of providing tokens this way is so that memory isn't consumed with a massive array (we don't care if the user decides to do this, it's their funeral :), and then they can look at the tokens and find out stuff.

The issue is that they may want the highlighted source also. So that has to be supported.

And also, it might be quite cool if, given an array of tokens and a new renderer, output for that renderer and tokens could be made. Although that's a separate issue really.